[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:16 dist-test-slave-2j90 krb5kdc[18826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:16 dist-test-slave-2j90 krb5kdc[18826](info): set up 2 sockets
May 01 02:02:16 dist-test-slave-2j90 krb5kdc[18826](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260501 02:02:18.122275 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.926s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:18.257627 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44451
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:18.351224 18842 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:18.351468 18842 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:18.351550 18842 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:18.354686 18842 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:18.354770 18842 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:18.354813 18842 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:18.354872 18842 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:18.354913 18842 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:18.358872 18842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:18.359736 18842 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:18.360498 18842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:18.365585 18847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:18.365733 18842 server_base.cc:1061] running on GCE node
W20260501 02:02:18.365633 18850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:18.365585 18848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:18.366436 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:18.367214 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:18.368417 18842 hybrid_clock.cc:648] HybridClock initialized: now 1777600938368398 us; error 27 us; skew 500 ppm
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:18.370950 18842 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:18.371846 18842 webserver.cc:492] Webserver started at http://127.18.96.254:43155/ using document root <none> and password file <none>
I20260501 02:02:18.372334 18842 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:18.372431 18842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:18.372634 18842 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:18.374258 18842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "53920c0b759843eb8ab40b8dcccd7c76"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "ff7f3bb8a97d9b5a5578611307f6baf4"
server_key_iv: "0b758e75c8567c963e964d69cb4c7137"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.374725 18842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "53920c0b759843eb8ab40b8dcccd7c76"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "ff7f3bb8a97d9b5a5578611307f6baf4"
server_key_iv: "0b758e75c8567c963e964d69cb4c7137"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.377748 18842 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:02:18.379710 18857 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.380534 18842 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:18.380688 18842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "53920c0b759843eb8ab40b8dcccd7c76"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "ff7f3bb8a97d9b5a5578611307f6baf4"
server_key_iv: "0b758e75c8567c963e964d69cb4c7137"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.380784 18842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:18.388778 18842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:18.390926 18842 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:18.391103 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:18.397756 18842 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44451
I20260501 02:02:18.397773 18909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44451 every 8 connection(s)
I20260501 02:02:18.398705 18842 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:18.401202 18910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:18.402812 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 18842
I20260501 02:02:18.402954 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:18.404189 18819 external_mini_cluster.cc:1468] Setting key d55511928357b1707f524b392ddc90de
I20260501 02:02:18.407568 18910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76: Bootstrap starting.
I20260501 02:02:18.409680 18910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:18.410524 18910 log.cc:826] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:18.412297 18910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76: No bootstrap required, opened a new log
I20260501 02:02:18.415083 18910 raft_consensus.cc:359] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } }
I20260501 02:02:18.415261 18910 raft_consensus.cc:385] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:18.415293 18910 raft_consensus.cc:740] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53920c0b759843eb8ab40b8dcccd7c76, State: Initialized, Role: FOLLOWER
I20260501 02:02:18.415724 18910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } }
I20260501 02:02:18.415874 18910 raft_consensus.cc:399] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:18.415951 18910 raft_consensus.cc:493] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:18.416038 18910 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 0 FOLLOWER]: Advancing to term 1
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:18.416934 18910 raft_consensus.cc:515] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } }
I20260501 02:02:18.417285 18910 leader_election.cc:304] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53920c0b759843eb8ab40b8dcccd7c76; no voters:
I20260501 02:02:18.417586 18910 leader_election.cc:290] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:18.417711 18915 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:18.417955 18915 raft_consensus.cc:697] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [term 1 LEADER]: Becoming Leader. State: Replica: 53920c0b759843eb8ab40b8dcccd7c76, State: Running, Role: LEADER
I20260501 02:02:18.418321 18915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } }
I20260501 02:02:18.418805 18910 sys_catalog.cc:565] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:18.419816 18916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53920c0b759843eb8ab40b8dcccd7c76. Latest consensus state: current_term: 1 leader_uuid: "53920c0b759843eb8ab40b8dcccd7c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } } }
I20260501 02:02:18.419931 18916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:18.420127 18916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53920c0b759843eb8ab40b8dcccd7c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53920c0b759843eb8ab40b8dcccd7c76" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44451 } } }
I20260501 02:02:18.420198 18916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:18.421033 18924 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:18.423089 18913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:18.406670 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.18.96.254:44451)
0501 02:02:18.407128 (+ 458us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:18.407139 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:18.407375 (+ 236us) server_negotiation.cc:408] Connection header received
0501 02:02:18.409601 (+ 2226us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:18.409626 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:18.409990 (+ 364us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:18.410292 (+ 302us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:18.412020 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.413101 (+ 1081us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:18.413797 (+ 696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.414078 (+ 281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:18.417151 (+ 3073us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:18.417171 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:18.417183 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:18.417212 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:18.420105 (+ 2893us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.420707 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.420712 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.420717 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.420824 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.421153 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.421156 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.421158 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.421512 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:18.421690 (+ 178us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:18.422246 (+ 556us) server_negotiation.cc:300] Negotiation successful
0501 02:02:18.422460 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":106,"threads_started":1}
I20260501 02:02:18.423794 18924 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:18.428215 18924 catalog_manager.cc:1357] Generated new cluster ID: c9c41779a1a14780b38023a788a25e4e
I20260501 02:02:18.428274 18924 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:18.439064 18924 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:18.439872 18924 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:18.448201 18924 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 53920c0b759843eb8ab40b8dcccd7c76: Generated new TSK 0
I20260501 02:02:18.448637 18924 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:18.497347 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44451
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:18.593111 18938 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:18.593302 18938 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:18.593356 18938 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:18.596603 18938 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:18.596653 18938 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:18.596733 18938 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:18.600808 18938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:18.602005 18938 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:18.602815 18938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 02:02:18.608952 18938 server_base.cc:1061] running on GCE node
W20260501 02:02:18.608906 18946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:18.608891 18944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:18.608899 18943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:18.609512 18938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:18.610038 18938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:18.611183 18938 hybrid_clock.cc:648] HybridClock initialized: now 1777600938611150 us; error 46 us; skew 500 ppm
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:18.613492 18938 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:18.614496 18938 webserver.cc:492] Webserver started at http://127.18.96.193:35053/ using document root <none> and password file <none>
I20260501 02:02:18.614995 18938 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:18.615042 18938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:18.615226 18938 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:18.616688 18938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "32bc0c7031a946d0811756387342599c"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "8d66d1c840a817dd545e115044231525"
server_key_iv: "c99ffab2f3df4265f61618105c73aafc"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.617120 18938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "32bc0c7031a946d0811756387342599c"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "8d66d1c840a817dd545e115044231525"
server_key_iv: "c99ffab2f3df4265f61618105c73aafc"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.620149 18938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:02:18.622105 18953 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.622928 18938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:18.623030 18938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "32bc0c7031a946d0811756387342599c"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "8d66d1c840a817dd545e115044231525"
server_key_iv: "c99ffab2f3df4265f61618105c73aafc"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.623116 18938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:18.633414 18938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:18.635826 18938 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:18.635965 18938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:18.636407 18938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:18.637157 18938 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:18.637202 18938 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.637265 18938 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:18.637295 18938 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.646266 18938 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:36939
I20260501 02:02:18.646291 19066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:36939 every 8 connection(s)
I20260501 02:02:18.647080 18938 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:18.651759 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 18938
I20260501 02:02:18.651858 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:18.652058 18819 external_mini_cluster.cc:1468] Setting key a74cfbe26a823df77e743b7a6e093f0f
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:18.658773 18913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:18.648512 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:36469 (local address 127.18.96.254:44451)
0501 02:02:18.648680 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:18.648685 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:18.649298 (+ 613us) server_negotiation.cc:408] Connection header received
0501 02:02:18.650088 (+ 790us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:18.650091 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:18.650139 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:18.650211 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:18.651597 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.652357 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:18.653090 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.653282 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:18.655526 (+ 2244us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:18.655547 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:18.655550 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:18.655571 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:18.656957 (+ 1386us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.657422 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.657427 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.657432 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.657481 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.657768 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.657771 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.657773 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.657905 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:18.658044 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:18.658478 (+ 434us) server_negotiation.cc:300] Negotiation successful
0501 02:02:18.658595 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:02:18.659216 19069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:18.648793 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44451 (local address 127.18.96.193:36469)
0501 02:02:18.649172 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:18.649201 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:18.649855 (+ 654us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:18.650340 (+ 485us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:18.650347 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:18.650661 (+ 314us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:18.651445 (+ 784us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:18.651457 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.652499 (+ 1042us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:18.652503 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:18.652991 (+ 488us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:18.653002 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.653148 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:18.653699 (+ 551us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:18.653715 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:18.655379 (+ 1664us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:18.657088 (+ 1709us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:18.657094 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:18.657104 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:18.657320 (+ 216us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:18.657584 (+ 264us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:18.657587 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:18.657588 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:18.657666 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:18.658045 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:18.658051 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:18.658228 (+ 177us) client_negotiation.cc:770] Sending connection context
0501 02:02:18.658452 (+ 224us) client_negotiation.cc:241] Negotiation successful
0501 02:02:18.658647 (+ 195us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260501 02:02:18.660238 19067 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44451
I20260501 02:02:18.660449 19067 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:18.660982 19067 heartbeater.cc:507] Master 127.18.96.254:44451 requested a full tablet report, sending...
I20260501 02:02:18.662361 18874 ts_manager.cc:194] Registered new tserver with Master: 32bc0c7031a946d0811756387342599c (127.18.96.193:36939)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:02:18.663630 18874 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:36469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:18.696786 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44451
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:18.796128 19074 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:18.796295 19074 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:18.796347 19074 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:18.799418 19074 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:18.799463 19074 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:18.799536 19074 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:18.803561 19074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:18.804351 19074 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:18.805039 19074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:18.811161 19082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:18.811269 19079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:18.811286 19080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:18.811496 19074 server_base.cc:1061] running on GCE node
I20260501 02:02:18.811803 19074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:18.812290 19074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:18.813469 19074 hybrid_clock.cc:648] HybridClock initialized: now 1777600938813456 us; error 46 us; skew 500 ppm
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:18.815977 19074 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:18.816880 19074 webserver.cc:492] Webserver started at http://127.18.96.194:42813/ using document root <none> and password file <none>
I20260501 02:02:18.817382 19074 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:18.817462 19074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:18.817663 19074 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:18.819380 19074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "ff238ff15aa843f2a88f083328236b44"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "964ee2659c106e0f7855940721f894c6"
server_key_iv: "3d376a2fe9398a1a3bcbb3d3c5728652"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.820007 19074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "ff238ff15aa843f2a88f083328236b44"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "964ee2659c106e0f7855940721f894c6"
server_key_iv: "3d376a2fe9398a1a3bcbb3d3c5728652"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.823144 19074 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:02:18.825038 19089 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.825893 19074 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:18.826045 19074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "ff238ff15aa843f2a88f083328236b44"
format_stamp: "Formatted at 2026-05-01 02:02:18 on dist-test-slave-2j90"
server_key: "964ee2659c106e0f7855940721f894c6"
server_key_iv: "3d376a2fe9398a1a3bcbb3d3c5728652"
server_key_version: "encryptionkey@0"
I20260501 02:02:18.826149 19074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:18.836678 19074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:18.839212 19074 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:18.839401 19074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:18.839892 19074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:18.840664 19074 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:18.840735 19074 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.840801 19074 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:18.840849 19074 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:18.849489 19074 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:38829
I20260501 02:02:18.849524 19202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:38829 every 8 connection(s)
I20260501 02:02:18.850386 19074 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:18.851382 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19074
I20260501 02:02:18.851500 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:18.851724 18819 external_mini_cluster.cc:1468] Setting key bc64c84fb63a4425527fbe2d0bd2beec
May 01 02:02:18 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600938, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:18.862298 18913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:18.851956 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55209 (local address 127.18.96.254:44451)
0501 02:02:18.852104 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:18.852109 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:18.852800 (+ 691us) server_negotiation.cc:408] Connection header received
0501 02:02:18.853867 (+ 1067us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:18.853874 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:18.853936 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:18.854040 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:18.855381 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.856154 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:18.856761 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.856979 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:18.858953 (+ 1974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:18.858975 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:18.858982 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:18.859016 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:18.860609 (+ 1593us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.861104 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.861108 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.861109 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.861167 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:18.861410 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:18.861412 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:18.861413 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:18.861539 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:18.861609 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:18.862089 (+ 480us) server_negotiation.cc:300] Negotiation successful
0501 02:02:18.862182 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:02:18.862917 19205 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:18.852244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44451 (local address 127.18.96.194:55209)
0501 02:02:18.852646 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:18.852691 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:18.853672 (+ 981us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:18.854153 (+ 481us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:18.854160 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:18.854471 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:18.855239 (+ 768us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:18.855251 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.856278 (+ 1027us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:18.856281 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:18.856646 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:18.856651 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:18.856787 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:18.857301 (+ 514us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:18.857319 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:18.858808 (+ 1489us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:18.860729 (+ 1921us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:18.860733 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:18.860740 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:18.861003 (+ 263us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:18.861262 (+ 259us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:18.861264 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:18.861266 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:18.861343 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:18.861618 (+ 275us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:18.861622 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:18.861835 (+ 213us) client_negotiation.cc:770] Sending connection context
0501 02:02:18.862039 (+ 204us) client_negotiation.cc:241] Negotiation successful
0501 02:02:18.862196 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:18.863956 19203 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44451
I20260501 02:02:18.864187 19203 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:18.864724 19203 heartbeater.cc:507] Master 127.18.96.254:44451 requested a full tablet report, sending...
I20260501 02:02:18.865787 18874 ts_manager.cc:194] Registered new tserver with Master: ff238ff15aa843f2a88f083328236b44 (127.18.96.194:38829)
I20260501 02:02:18.866369 18874 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:55209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:18.897482 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44451
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:18.995019 19210 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:18.995182 19210 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:18.995247 19210 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:18.998312 19210 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:18.998358 19210 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:18.998449 19210 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:19.002468 19210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44451
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:19.003316 19210 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:19.004017 19210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:19.009761 19216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:19.009877 19218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:19.010033 19210 server_base.cc:1061] running on GCE node
W20260501 02:02:19.009943 19215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:19.010476 19210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:19.010941 19210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:19.012115 19210 hybrid_clock.cc:648] HybridClock initialized: now 1777600939012107 us; error 27 us; skew 500 ppm
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:19.014708 19210 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:19.015684 19210 webserver.cc:492] Webserver started at http://127.18.96.195:44413/ using document root <none> and password file <none>
I20260501 02:02:19.016134 19210 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:19.016176 19210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:19.016314 19210 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:19.017802 19210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "7d362069cf34461092b5d00f82d803ef"
format_stamp: "Formatted at 2026-05-01 02:02:19 on dist-test-slave-2j90"
server_key: "d810970312ba4c075357a44018cedf34"
server_key_iv: "fe10b036307acc2456c31ffdb59db325"
server_key_version: "encryptionkey@0"
I20260501 02:02:19.018270 19210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "7d362069cf34461092b5d00f82d803ef"
format_stamp: "Formatted at 2026-05-01 02:02:19 on dist-test-slave-2j90"
server_key: "d810970312ba4c075357a44018cedf34"
server_key_iv: "fe10b036307acc2456c31ffdb59db325"
server_key_version: "encryptionkey@0"
I20260501 02:02:19.021234 19210 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:19.023151 19225 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:19.024119 19210 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:19.024240 19210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "7d362069cf34461092b5d00f82d803ef"
format_stamp: "Formatted at 2026-05-01 02:02:19 on dist-test-slave-2j90"
server_key: "d810970312ba4c075357a44018cedf34"
server_key_iv: "fe10b036307acc2456c31ffdb59db325"
server_key_version: "encryptionkey@0"
I20260501 02:02:19.024326 19210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:19.034832 19210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:19.037421 19210 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:19.037604 19210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:19.038194 19210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:19.039004 19210 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:19.039049 19210 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:19.039116 19210 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:19.039158 19210 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:19.047509 19210 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:42695
I20260501 02:02:19.047535 19338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:42695 every 8 connection(s)
I20260501 02:02:19.048339 19210 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:19.052115 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19210
I20260501 02:02:19.052218 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:19.052434 18819 external_mini_cluster.cc:1468] Setting key f23abd293890662d797d8e6a32e4f51e
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:19.060225 18913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:19.049986 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:58283 (local address 127.18.96.254:44451)
0501 02:02:19.050122 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:19.050126 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:19.050742 (+ 616us) server_negotiation.cc:408] Connection header received
0501 02:02:19.051527 (+ 785us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:19.051530 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:19.051590 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:19.051691 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:19.053231 (+ 1540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.053947 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:19.054722 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.054889 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:19.057036 (+ 2147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:19.057049 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:19.057070 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:19.057094 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:19.058474 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.058907 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.058910 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.058914 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.058958 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.059277 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.059280 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.059282 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.059417 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:19.059524 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:19.059964 (+ 440us) server_negotiation.cc:300] Negotiation successful
0501 02:02:19.060072 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:02:19.060770 19341 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:19.050219 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44451 (local address 127.18.96.195:58283)
0501 02:02:19.050616 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:19.050644 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:19.051335 (+ 691us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:19.051844 (+ 509us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:19.051851 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:19.052162 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:19.053099 (+ 937us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:19.053117 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.054130 (+ 1013us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:19.054133 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:19.054608 (+ 475us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:19.054615 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.054762 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:19.055281 (+ 519us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:19.055298 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:19.056884 (+ 1586us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:19.058593 (+ 1709us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:19.058598 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:19.058607 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:19.058811 (+ 204us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:19.059063 (+ 252us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:19.059066 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:19.059068 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:19.059160 (+ 92us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:19.059533 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:19.059541 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:19.059743 (+ 202us) client_negotiation.cc:770] Sending connection context
0501 02:02:19.059915 (+ 172us) client_negotiation.cc:241] Negotiation successful
0501 02:02:19.060107 (+ 192us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260501 02:02:19.061722 19339 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44451
I20260501 02:02:19.061926 19339 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:19.062371 19339 heartbeater.cc:507] Master 127.18.96.254:44451 requested a full tablet report, sending...
I20260501 02:02:19.063249 18874 ts_manager.cc:194] Registered new tserver with Master: 7d362069cf34461092b5d00f82d803ef (127.18.96.195:42695)
I20260501 02:02:19.063755 18874 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:58283
I20260501 02:02:19.066346 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:19.089821 19346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:19.081688 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.18.96.193:36939)
0501 02:02:19.082022 (+ 334us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:19.082031 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:19.082069 (+ 38us) server_negotiation.cc:408] Connection header received
0501 02:02:19.082129 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:19.082132 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:19.082298 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:19.082463 (+ 165us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:19.083250 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.083923 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:19.084568 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.084856 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:19.086757 (+ 1901us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:19.086799 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:19.086808 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:19.086831 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:19.088267 (+ 1436us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.088682 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.088687 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.088689 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.088736 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.088969 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.088973 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.088975 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.089131 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:19.089324 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:19.089531 (+ 207us) server_negotiation.cc:300] Negotiation successful
0501 02:02:19.089667 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
W20260501 02:02:19.090700 18981 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49800
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[18826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:19.113025 19346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:19.106158 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.18.96.193:36939)
0501 02:02:19.106319 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:19.106323 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:19.106403 (+ 80us) server_negotiation.cc:408] Connection header received
0501 02:02:19.106509 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:19.106512 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:19.106557 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:19.106627 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:19.107349 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.107828 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:19.108426 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:19.108571 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:19.110609 (+ 2038us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:19.110624 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:19.110626 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:19.110647 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:19.111796 (+ 1149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.112127 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.112129 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.112130 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.112165 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:19.112327 (+ 162us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:19.112329 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:19.112330 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:19.112423 (+ 93us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:19.112556 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:19.112741 (+ 185us) server_negotiation.cc:300] Negotiation successful
0501 02:02:19.112821 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:02:19.113991 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 18938
I20260501 02:02:19.120481 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19074
I20260501 02:02:19.129138 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19210
I20260501 02:02:19.136349 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 18842
2026-05-01T02:02:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2968 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[19353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[19353](info): set up 2 sockets
May 01 02:02:19 dist-test-slave-2j90 krb5kdc[19353](info): commencing operation
krb5kdc: starting...
W20260501 02:02:21.090241 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:21.224892 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44823
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:21.322360 19369 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:21.322538 19369 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:21.322594 19369 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:21.325599 19369 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:21.325644 19369 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:21.325664 19369 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:21.325681 19369 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:21.325698 19369 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:21.329455 19369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:21.330250 19369 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:21.330906 19369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:21.335841 19375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.335922 19374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.336007 19377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:21.336198 19369 server_base.cc:1061] running on GCE node
I20260501 02:02:21.336592 19369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:21.337405 19369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:21.338580 19369 hybrid_clock.cc:648] HybridClock initialized: now 1777600941338552 us; error 47 us; skew 500 ppm
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:21.341014 19369 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:21.341946 19369 webserver.cc:492] Webserver started at http://127.18.96.254:34953/ using document root <none> and password file <none>
I20260501 02:02:21.342505 19369 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:21.342577 19369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:21.342747 19369 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:21.344210 19369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "7f346227e5bb4ec99d50610f75263090"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "6f57a1835a7b0a4c096a21744b2720fa"
server_key_iv: "70d6dc4cc84b27f10ebc06dfca637b43"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.344663 19369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "7f346227e5bb4ec99d50610f75263090"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "6f57a1835a7b0a4c096a21744b2720fa"
server_key_iv: "70d6dc4cc84b27f10ebc06dfca637b43"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.347582 19369 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:21.349550 19384 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.350406 19369 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:21.350528 19369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "7f346227e5bb4ec99d50610f75263090"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "6f57a1835a7b0a4c096a21744b2720fa"
server_key_iv: "70d6dc4cc84b27f10ebc06dfca637b43"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.350620 19369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:21.367642 19369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:21.373466 19369 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:21.373648 19369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:21.383013 19436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44823 every 8 connection(s)
I20260501 02:02:21.383014 19369 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44823
I20260501 02:02:21.384207 19369 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:21.387063 19437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:21.389554 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19369
I20260501 02:02:21.389644 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:21.389860 18819 external_mini_cluster.cc:1468] Setting key 457d8ba97051206623400b5e610d0ad0
I20260501 02:02:21.392541 19437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: Bootstrap starting.
I20260501 02:02:21.394559 19437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:21.395296 19437 log.cc:826] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: Log is configured to *not* fsync() on all Append() calls
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:21.397084 19437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: No bootstrap required, opened a new log
I20260501 02:02:21.399724 19437 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } }
I20260501 02:02:21.399921 19437 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:21.399984 19437 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f346227e5bb4ec99d50610f75263090, State: Initialized, Role: FOLLOWER
I20260501 02:02:21.400465 19437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } }
I20260501 02:02:21.400625 19437 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:21.400691 19437 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:21.400782 19437 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:21.401804 19437 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } }
I20260501 02:02:21.402048 19440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:21.391124 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.18.96.254:44823)
0501 02:02:21.391499 (+ 375us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:21.391507 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:21.391536 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:02:21.392000 (+ 464us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:21.392014 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:21.392257 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:21.392555 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:21.393412 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.394236 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:21.394945 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.395228 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:21.397540 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:21.397557 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:21.397574 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:21.397600 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:21.399349 (+ 1749us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.399780 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.399784 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.399788 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.399856 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.400138 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.400142 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.400144 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.400535 (+ 391us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:21.400751 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:21.401085 (+ 334us) server_negotiation.cc:300] Negotiation successful
0501 02:02:21.401298 (+ 213us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":123,"threads_started":1}
I20260501 02:02:21.402168 19437 leader_election.cc:304] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f346227e5bb4ec99d50610f75263090; no voters:
I20260501 02:02:21.402467 19437 leader_election.cc:290] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:21.402559 19442 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:21.402743 19442 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [term 1 LEADER]: Becoming Leader. State: Replica: 7f346227e5bb4ec99d50610f75263090, State: Running, Role: LEADER
I20260501 02:02:21.402997 19442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } }
I20260501 02:02:21.403470 19437 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:21.404064 19444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f346227e5bb4ec99d50610f75263090. Latest consensus state: current_term: 1 leader_uuid: "7f346227e5bb4ec99d50610f75263090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } } }
I20260501 02:02:21.404183 19444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:21.404454 19443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f346227e5bb4ec99d50610f75263090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f346227e5bb4ec99d50610f75263090" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44823 } } }
I20260501 02:02:21.404544 19443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090 [sys.catalog]: This master's current role is: LEADER
W20260501 02:02:21.408152 19457 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:02:21.408229 19457 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:02:21.408281 19447 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:21.409155 19447 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:21.413659 19447 catalog_manager.cc:1357] Generated new cluster ID: b27ab73d120f49f98ef1302c54a2b542
I20260501 02:02:21.413731 19447 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:21.420622 19447 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:21.421613 19447 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:21.430969 19447 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7f346227e5bb4ec99d50610f75263090: Generated new TSK 0
I20260501 02:02:21.431511 19447 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:21.486522 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44823
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:21.582170 19465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:21.582376 19465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:21.582463 19465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:21.585609 19465 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:21.585685 19465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:21.585794 19465 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:21.589740 19465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:21.590647 19465 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:21.591372 19465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:21.597180 19473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.597259 19471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.597180 19470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:21.597446 19465 server_base.cc:1061] running on GCE node
I20260501 02:02:21.597740 19465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:21.598232 19465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:21.599378 19465 hybrid_clock.cc:648] HybridClock initialized: now 1777600941599354 us; error 37 us; skew 500 ppm
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:21.601684 19465 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:21.602649 19465 webserver.cc:492] Webserver started at http://127.18.96.193:44579/ using document root <none> and password file <none>
I20260501 02:02:21.603137 19465 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:21.603205 19465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:21.603389 19465 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:21.605022 19465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "b61962f1608c4ca3aaedf2f2dac3bf79"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "9a5a4a65b78869099e2a8968d9d818d2"
server_key_iv: "0f817f2ecb729beee1c73229f109b8bf"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.605466 19465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "b61962f1608c4ca3aaedf2f2dac3bf79"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "9a5a4a65b78869099e2a8968d9d818d2"
server_key_iv: "0f817f2ecb729beee1c73229f109b8bf"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.608556 19465 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260501 02:02:21.610411 19480 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.611270 19465 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:21.611352 19465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "b61962f1608c4ca3aaedf2f2dac3bf79"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "9a5a4a65b78869099e2a8968d9d818d2"
server_key_iv: "0f817f2ecb729beee1c73229f109b8bf"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.611416 19465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:21.622509 19465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:21.624567 19465 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:21.624701 19465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:21.625097 19465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:21.625846 19465 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:21.625890 19465 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.625953 19465 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:21.626005 19465 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.634789 19465 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44761
I20260501 02:02:21.634828 19593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44761 every 8 connection(s)
I20260501 02:02:21.635624 19465 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:21.641342 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19465
I20260501 02:02:21.641522 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:21.641774 18819 external_mini_cluster.cc:1468] Setting key b070604f9da24323b400a342f3f232f8
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:21.646842 19440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:21.637104 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:34579 (local address 127.18.96.254:44823)
0501 02:02:21.637227 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:21.637230 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:21.637794 (+ 564us) server_negotiation.cc:408] Connection header received
0501 02:02:21.638583 (+ 789us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:21.638586 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:21.638634 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:21.638706 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:21.640016 (+ 1310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.640522 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:21.641100 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.641317 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:21.643342 (+ 2025us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:21.643355 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:21.643358 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:21.643378 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:21.644779 (+ 1401us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.645350 (+ 571us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.645356 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.645361 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.645415 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.645681 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.645687 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.645691 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.645990 (+ 299us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:21.646095 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:21.646482 (+ 387us) server_negotiation.cc:300] Negotiation successful
0501 02:02:21.646606 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260501 02:02:21.647207 19596 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:21.637269 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44823 (local address 127.18.96.193:34579)
0501 02:02:21.637662 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:21.637690 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:21.638396 (+ 706us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:21.638830 (+ 434us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:21.638838 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:21.639138 (+ 300us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:21.639879 (+ 741us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:21.639890 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.640639 (+ 749us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:21.640641 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:21.640996 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:21.641002 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.641130 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:21.641691 (+ 561us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:21.641708 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:21.643177 (+ 1469us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:21.644895 (+ 1718us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:21.644903 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:21.644914 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:21.645239 (+ 325us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:21.645503 (+ 264us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:21.645505 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:21.645507 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:21.645588 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:21.646096 (+ 508us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:21.646099 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:21.646259 (+ 160us) client_negotiation.cc:770] Sending connection context
0501 02:02:21.646403 (+ 144us) client_negotiation.cc:241] Negotiation successful
0501 02:02:21.646589 (+ 186us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260501 02:02:21.648227 19594 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44823
I20260501 02:02:21.648468 19594 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:21.648900 19594 heartbeater.cc:507] Master 127.18.96.254:44823 requested a full tablet report, sending...
I20260501 02:02:21.650292 19401 ts_manager.cc:194] Registered new tserver with Master: b61962f1608c4ca3aaedf2f2dac3bf79 (127.18.96.193:44761)
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:21.651355 19401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:34579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:21.686905 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44823
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:21.781734 19601 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:21.781919 19601 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:21.781989 19601 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:21.784978 19601 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:21.785024 19601 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:21.785095 19601 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:21.789016 19601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:21.789857 19601 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:21.790581 19601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:21.796618 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.796677 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:21.796629 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:21.796778 19601 server_base.cc:1061] running on GCE node
I20260501 02:02:21.797159 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:21.797658 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:21.798825 19601 hybrid_clock.cc:648] HybridClock initialized: now 1777600941798808 us; error 42 us; skew 500 ppm
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:21.801302 19601 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:21.802323 19601 webserver.cc:492] Webserver started at http://127.18.96.194:40639/ using document root <none> and password file <none>
I20260501 02:02:21.802852 19601 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:21.802924 19601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:21.803115 19601 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:21.804706 19601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "006192a2cf6e4c26a2add61739db10f6"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "892bb5edc24f4686910d8af451ca220d"
server_key_iv: "9215859cf13ac7a2f90d68f56706c496"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.805171 19601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "006192a2cf6e4c26a2add61739db10f6"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "892bb5edc24f4686910d8af451ca220d"
server_key_iv: "9215859cf13ac7a2f90d68f56706c496"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.808276 19601 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:02:21.810191 19616 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.811075 19601 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:02:21.811192 19601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "006192a2cf6e4c26a2add61739db10f6"
format_stamp: "Formatted at 2026-05-01 02:02:21 on dist-test-slave-2j90"
server_key: "892bb5edc24f4686910d8af451ca220d"
server_key_iv: "9215859cf13ac7a2f90d68f56706c496"
server_key_version: "encryptionkey@0"
I20260501 02:02:21.811301 19601 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:21.824210 19601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:21.826750 19601 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:21.826944 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:21.827433 19601 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:21.828251 19601 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:21.828323 19601 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.828392 19601 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:21.828436 19601 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:21.837179 19601 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:35545
I20260501 02:02:21.837206 19729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:35545 every 8 connection(s)
I20260501 02:02:21.838065 19601 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:21.841620 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19601
I20260501 02:02:21.841739 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:21.842015 18819 external_mini_cluster.cc:1468] Setting key a3019fc7e8656cacbb27a0de7be00827
May 01 02:02:21 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:21.850737 19440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:21.839667 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40437 (local address 127.18.96.254:44823)
0501 02:02:21.839820 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:21.839825 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:21.840453 (+ 628us) server_negotiation.cc:408] Connection header received
0501 02:02:21.841190 (+ 737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:21.841193 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:21.841239 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:21.841314 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:21.843247 (+ 1933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.843767 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:21.844411 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.844591 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:21.847107 (+ 2516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:21.847122 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:21.847124 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:21.847147 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:21.848531 (+ 1384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.849298 (+ 767us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.849303 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.849304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.849354 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:21.849715 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:21.849718 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:21.849719 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:21.849856 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:21.849931 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:21.850528 (+ 597us) server_negotiation.cc:300] Negotiation successful
0501 02:02:21.850620 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:21.851508 19732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:21.839952 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44823 (local address 127.18.96.194:40437)
0501 02:02:21.840324 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:21.840354 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:21.841027 (+ 673us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:21.841523 (+ 496us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:21.841535 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:21.842068 (+ 533us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:21.843092 (+ 1024us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:21.843107 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.843903 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:21.843909 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:21.844294 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:21.844300 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:21.844495 (+ 195us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:21.845081 (+ 586us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:21.845110 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:21.846945 (+ 1835us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:21.848664 (+ 1719us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:21.848671 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:21.848683 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:21.849032 (+ 349us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:21.849479 (+ 447us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:21.849485 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:21.849489 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:21.849611 (+ 122us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:21.849990 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:21.849995 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:21.850274 (+ 279us) client_negotiation.cc:770] Sending connection context
0501 02:02:21.850509 (+ 235us) client_negotiation.cc:241] Negotiation successful
0501 02:02:21.850740 (+ 231us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260501 02:02:21.852511 19730 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44823
I20260501 02:02:21.852738 19730 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:21.853232 19730 heartbeater.cc:507] Master 127.18.96.254:44823 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:21.854466 19401 ts_manager.cc:194] Registered new tserver with Master: 006192a2cf6e4c26a2add61739db10f6 (127.18.96.194:35545)
I20260501 02:02:21.855085 19401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:40437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:21.889775 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44823
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:21.990777 19737 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:21.990947 19737 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:21.991057 19737 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:21.994634 19737 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:21.994686 19737 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:21.994803 19737 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:21.999526 19737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:22.000497 19737 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:22.001331 19737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:22.008092 19745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:22.008147 19742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:22.008100 19743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:22.008433 19737 server_base.cc:1061] running on GCE node
I20260501 02:02:22.008778 19737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:22.009302 19737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:22.010445 19737 hybrid_clock.cc:648] HybridClock initialized: now 1777600942010422 us; error 36 us; skew 500 ppm
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:22.013193 19737 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:22.014230 19737 webserver.cc:492] Webserver started at http://127.18.96.195:43035/ using document root <none> and password file <none>
I20260501 02:02:22.014739 19737 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:22.014822 19737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:22.015028 19737 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:22.016575 19737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "97f5ee1f45684a34a1bc84dba990ca4a"
format_stamp: "Formatted at 2026-05-01 02:02:22 on dist-test-slave-2j90"
server_key: "07a6934ec3976577315d2133b18b7365"
server_key_iv: "426ffcc2c7946e4edbf6b45d9a936ebf"
server_key_version: "encryptionkey@0"
I20260501 02:02:22.016983 19737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "97f5ee1f45684a34a1bc84dba990ca4a"
format_stamp: "Formatted at 2026-05-01 02:02:22 on dist-test-slave-2j90"
server_key: "07a6934ec3976577315d2133b18b7365"
server_key_iv: "426ffcc2c7946e4edbf6b45d9a936ebf"
server_key_version: "encryptionkey@0"
I20260501 02:02:22.020009 19737 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:22.021816 19752 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:22.022738 19737 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:22.022845 19737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "97f5ee1f45684a34a1bc84dba990ca4a"
format_stamp: "Formatted at 2026-05-01 02:02:22 on dist-test-slave-2j90"
server_key: "07a6934ec3976577315d2133b18b7365"
server_key_iv: "426ffcc2c7946e4edbf6b45d9a936ebf"
server_key_version: "encryptionkey@0"
I20260501 02:02:22.022935 19737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:22.039072 19737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:22.041730 19737 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:22.041931 19737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:22.042471 19737 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:22.043270 19737 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:22.043315 19737 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:22.043380 19737 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:22.043417 19737 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:22.052196 19737 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:41547
I20260501 02:02:22.052234 19865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:41547 every 8 connection(s)
I20260501 02:02:22.053021 19737 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:22.054592 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19737
I20260501 02:02:22.054711 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:22.054965 18819 external_mini_cluster.cc:1468] Setting key 2d8cb964e9bd4f5d1b770b199ba1594f
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:22.064577 19440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.054755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:54713 (local address 127.18.96.254:44823)
0501 02:02:22.054888 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.054892 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.055701 (+ 809us) server_negotiation.cc:408] Connection header received
0501 02:02:22.056692 (+ 991us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.056694 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.056739 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.056807 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:22.058241 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.058743 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.059292 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.059419 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.061484 (+ 2065us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:22.061509 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:22.061511 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:22.061532 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:22.062824 (+ 1292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.063282 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.063288 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.063290 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.063340 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.063634 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.063643 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.063644 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.063767 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:22.063883 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.064317 (+ 434us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.064424 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:02:22.065141 19868 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.055068 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44823 (local address 127.18.96.195:54713)
0501 02:02:22.055544 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.055617 (+ 73us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.056494 (+ 877us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.056954 (+ 460us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.056966 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.057308 (+ 342us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:22.058117 (+ 809us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.058130 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.058838 (+ 708us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.058841 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.059181 (+ 340us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.059186 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.059311 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.059827 (+ 516us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:22.059842 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:22.061330 (+ 1488us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:22.062944 (+ 1614us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:22.062950 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:22.062959 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:22.063177 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:22.063431 (+ 254us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:22.063434 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:22.063437 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:22.063525 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:22.063890 (+ 365us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:22.063898 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:22.064092 (+ 194us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.064267 (+ 175us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.064459 (+ 192us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":120,"threads_started":1}
I20260501 02:02:22.066141 19866 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44823
I20260501 02:02:22.066354 19866 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:22.066781 19866 heartbeater.cc:507] Master 127.18.96.254:44823 requested a full tablet report, sending...
I20260501 02:02:22.067754 19401 ts_manager.cc:194] Registered new tserver with Master: 97f5ee1f45684a34a1bc84dba990ca4a (127.18.96.195:41547)
I20260501 02:02:22.068276 19401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:54713
I20260501 02:02:22.068876 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:02:22.077497 19440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.071549 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.18.96.254:44823)
0501 02:02:22.071663 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.071666 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.071889 (+ 223us) server_negotiation.cc:408] Connection header received
0501 02:02:22.071970 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.071972 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.072007 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.072071 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:22.072783 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.073253 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.073847 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.074052 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.075069 (+ 1017us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:22.075081 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:22.075083 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:22.075099 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:22.076225 (+ 1126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.076652 (+ 427us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.076655 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.076656 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.076695 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.076888 (+ 193us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.076890 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.076892 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.077022 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:22.077119 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.077284 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.077371 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:02:22.081204 19400 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:22.083499 19400 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:22.095299 19878 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.091066 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35545 (local address 127.0.0.1:47364)
0501 02:02:22.091686 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.091721 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.091850 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.092494 (+ 644us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.092499 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.092521 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:22.092764 (+ 243us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.092770 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.093890 (+ 1120us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.093894 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.094809 (+ 915us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.094817 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.094926 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.094975 (+ 49us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.095052 (+ 77us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.095145 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"spinlock_wait_cycles":3840,"thread_start_us":86,"threads_started":1}
I20260501 02:02:22.095918 19877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.091357 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:41547 (local address 127.0.0.1:60906)
0501 02:02:22.092290 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.092303 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.092384 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.092866 (+ 482us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.092872 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.092894 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:22.093115 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.093121 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.094573 (+ 1452us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.094577 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.095557 (+ 980us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.095567 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.095693 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.095707 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.095756 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.095800 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":104,"threads_started":1}
I20260501 02:02:22.096196 19879 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.091205 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.18.96.194:35545)
0501 02:02:22.091549 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.091555 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.091800 (+ 245us) server_negotiation.cc:408] Connection header received
0501 02:02:22.092030 (+ 230us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.092035 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.092201 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.092327 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:22.092912 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.093764 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.095162 (+ 1398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.095780 (+ 618us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.095891 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.095957 (+ 66us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.096046 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":118,"threads_started":1}
I20260501 02:02:22.096217 19881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.091485 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.18.96.193:44761)
0501 02:02:22.092153 (+ 668us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.092159 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.092179 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:02:22.092256 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.092262 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.092393 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.092522 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:22.092939 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.094052 (+ 1113us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.095374 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.095861 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.095930 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.096005 (+ 75us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.096088 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":118,"threads_started":1}
I20260501 02:02:22.095299 19880 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.090927 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44761 (local address 127.0.0.1:37386)
0501 02:02:22.091686 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.091721 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.091845 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.092530 (+ 685us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.092532 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.092560 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:22.092811 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.092820 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.094145 (+ 1325us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.094149 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.094960 (+ 811us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.094967 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.095062 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.095073 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.095136 (+ 63us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.095176 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260501 02:02:22.096997 19882 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.091737 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.18.96.195:41547)
0501 02:02:22.092154 (+ 417us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.092159 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.092497 (+ 338us) server_negotiation.cc:408] Connection header received
0501 02:02:22.092572 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.092577 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.092715 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.092848 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:22.093225 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.094318 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.096146 (+ 1828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.096663 (+ 517us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.096741 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.096785 (+ 44us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.096834 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":114,"threads_started":1}
I20260501 02:02:22.098933 19664 tablet_service.cc:1511] Processing CreateTablet for tablet fc56502626ec4619ab4be5ad38a94137 (DEFAULT_TABLE table=test-table [id=85d6dc2fd4fb4adea12cd9d39abcf8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:22.099303 19800 tablet_service.cc:1511] Processing CreateTablet for tablet fc56502626ec4619ab4be5ad38a94137 (DEFAULT_TABLE table=test-table [id=85d6dc2fd4fb4adea12cd9d39abcf8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:22.099303 19528 tablet_service.cc:1511] Processing CreateTablet for tablet fc56502626ec4619ab4be5ad38a94137 (DEFAULT_TABLE table=test-table [id=85d6dc2fd4fb4adea12cd9d39abcf8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:22.099776 19664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc56502626ec4619ab4be5ad38a94137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:22.100136 19800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc56502626ec4619ab4be5ad38a94137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:22.100136 19528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc56502626ec4619ab4be5ad38a94137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:22.104864 19883 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: Bootstrap starting.
I20260501 02:02:22.105988 19884 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Bootstrap starting.
I20260501 02:02:22.106405 19885 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: Bootstrap starting.
I20260501 02:02:22.107739 19884 tablet_bootstrap.cc:654] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:22.107739 19883 tablet_bootstrap.cc:654] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:22.107957 19885 tablet_bootstrap.cc:654] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:22.108574 19885 log.cc:826] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:22.108563 19884 log.cc:826] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:22.108563 19883 log.cc:826] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:22.110239 19884 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: No bootstrap required, opened a new log
I20260501 02:02:22.110278 19883 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: No bootstrap required, opened a new log
I20260501 02:02:22.110278 19885 tablet_bootstrap.cc:492] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: No bootstrap required, opened a new log
I20260501 02:02:22.110451 19884 ts_tablet_manager.cc:1403] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260501 02:02:22.110451 19883 ts_tablet_manager.cc:1403] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:02:22.110451 19885 ts_tablet_manager.cc:1403] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:02:22.113301 19884 raft_consensus.cc:359] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.113488 19884 raft_consensus.cc:385] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:22.113547 19884 raft_consensus.cc:740] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61962f1608c4ca3aaedf2f2dac3bf79, State: Initialized, Role: FOLLOWER
I20260501 02:02:22.113881 19883 raft_consensus.cc:359] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.113907 19885 raft_consensus.cc:359] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.114013 19884 consensus_queue.cc:260] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.114105 19883 raft_consensus.cc:385] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:22.114105 19885 raft_consensus.cc:385] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:22.114166 19883 raft_consensus.cc:740] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006192a2cf6e4c26a2add61739db10f6, State: Initialized, Role: FOLLOWER
I20260501 02:02:22.114166 19885 raft_consensus.cc:740] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f5ee1f45684a34a1bc84dba990ca4a, State: Initialized, Role: FOLLOWER
I20260501 02:02:22.114571 19594 heartbeater.cc:499] Master 127.18.96.254:44823 was elected leader, sending a full tablet report...
I20260501 02:02:22.114578 19885 consensus_queue.cc:260] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.114578 19883 consensus_queue.cc:260] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.114916 19884 ts_tablet_manager.cc:1434] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
I20260501 02:02:22.115382 19885 ts_tablet_manager.cc:1434] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:02:22.115628 19730 heartbeater.cc:499] Master 127.18.96.254:44823 was elected leader, sending a full tablet report...
I20260501 02:02:22.115656 19866 heartbeater.cc:499] Master 127.18.96.254:44823 was elected leader, sending a full tablet report...
I20260501 02:02:22.115522 19883 ts_tablet_manager.cc:1434] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260501 02:02:22.136633 19595 tablet.cc:2404] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:22.154458 19889 raft_consensus.cc:493] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:22.154651 19889 raft_consensus.cc:515] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.155741 19889 leader_election.cc:290] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 006192a2cf6e4c26a2add61739db10f6 (127.18.96.194:35545), 97f5ee1f45684a34a1bc84dba990ca4a (127.18.96.195:41547)
I20260501 02:02:22.158591 19892 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.155891 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35545 (local address 127.18.96.193:34891)
0501 02:02:22.156183 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.156196 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.156300 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.156564 (+ 264us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.156567 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.156584 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:22.156813 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.156819 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.157547 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.157553 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.158177 (+ 624us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.158184 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.158319 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.158337 (+ 18us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.158391 (+ 54us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.158443 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260501 02:02:22.158758 19893 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.155959 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:41547 (local address 127.18.96.193:59869)
0501 02:02:22.156290 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:22.156300 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:22.156404 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:22.156675 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:22.156677 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:22.156694 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:22.156940 (+ 246us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.156945 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.157643 (+ 698us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.157648 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:22.158291 (+ 643us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:22.158298 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.158417 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.158435 (+ 18us) client_negotiation.cc:770] Sending connection context
0501 02:02:22.158485 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:02:22.158530 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260501 02:02:22.159144 19879 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.156090 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:34891 (local address 127.18.96.194:35545)
0501 02:02:22.156206 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.156212 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.156254 (+ 42us) server_negotiation.cc:408] Connection header received
0501 02:02:22.156413 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.156416 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.156455 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.156551 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:22.156937 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.157411 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.158291 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.158829 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.158865 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.158948 (+ 83us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.159013 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:02:22.159204 19882 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.156094 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:59869 (local address 127.18.96.195:41547)
0501 02:02:22.156237 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.156242 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.156366 (+ 124us) server_negotiation.cc:408] Connection header received
0501 02:02:22.156531 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.156534 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.156574 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.156663 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:22.157051 (+ 388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.157529 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.158404 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.158953 (+ 549us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.158987 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.159035 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.159106 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:02:22.159760 19684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc56502626ec4619ab4be5ad38a94137" candidate_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006192a2cf6e4c26a2add61739db10f6" is_pre_election: true
I20260501 02:02:22.159794 19820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc56502626ec4619ab4be5ad38a94137" candidate_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" is_pre_election: true
I20260501 02:02:22.160018 19684 raft_consensus.cc:2468] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61962f1608c4ca3aaedf2f2dac3bf79 in term 0.
I20260501 02:02:22.160025 19820 raft_consensus.cc:2468] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b61962f1608c4ca3aaedf2f2dac3bf79 in term 0.
I20260501 02:02:22.160454 19484 leader_election.cc:304] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006192a2cf6e4c26a2add61739db10f6, b61962f1608c4ca3aaedf2f2dac3bf79; no voters:
I20260501 02:02:22.160697 19889 raft_consensus.cc:2804] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:22.160764 19889 raft_consensus.cc:493] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:22.160804 19889 raft_consensus.cc:3060] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:22.161790 19889 raft_consensus.cc:515] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.162195 19889 leader_election.cc:290] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [CANDIDATE]: Term 1 election: Requested vote from peers 006192a2cf6e4c26a2add61739db10f6 (127.18.96.194:35545), 97f5ee1f45684a34a1bc84dba990ca4a (127.18.96.195:41547)
I20260501 02:02:22.162510 19684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc56502626ec4619ab4be5ad38a94137" candidate_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006192a2cf6e4c26a2add61739db10f6"
I20260501 02:02:22.162552 19820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc56502626ec4619ab4be5ad38a94137" candidate_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f5ee1f45684a34a1bc84dba990ca4a"
I20260501 02:02:22.162634 19684 raft_consensus.cc:3060] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:22.162669 19820 raft_consensus.cc:3060] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:22.163555 19684 raft_consensus.cc:2468] T fc56502626ec4619ab4be5ad38a94137 P 006192a2cf6e4c26a2add61739db10f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61962f1608c4ca3aaedf2f2dac3bf79 in term 1.
I20260501 02:02:22.163586 19820 raft_consensus.cc:2468] T fc56502626ec4619ab4be5ad38a94137 P 97f5ee1f45684a34a1bc84dba990ca4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61962f1608c4ca3aaedf2f2dac3bf79 in term 1.
I20260501 02:02:22.163908 19482 leader_election.cc:304] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97f5ee1f45684a34a1bc84dba990ca4a, b61962f1608c4ca3aaedf2f2dac3bf79; no voters:
I20260501 02:02:22.164098 19889 raft_consensus.cc:2804] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:22.164312 19889 raft_consensus.cc:697] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [term 1 LEADER]: Becoming Leader. State: Replica: b61962f1608c4ca3aaedf2f2dac3bf79, State: Running, Role: LEADER
I20260501 02:02:22.164633 19889 consensus_queue.cc:237] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } }
I20260501 02:02:22.167784 19401 catalog_manager.cc:5671] T fc56502626ec4619ab4be5ad38a94137 P b61962f1608c4ca3aaedf2f2dac3bf79 reported cstate change: term changed from 0 to 1, leader changed from <none> to b61962f1608c4ca3aaedf2f2dac3bf79 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61962f1608c4ca3aaedf2f2dac3bf79" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "006192a2cf6e4c26a2add61739db10f6" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97f5ee1f45684a34a1bc84dba990ca4a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 41547 } health_report { overall_health: UNKNOWN } } }
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:22.193634 19881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.186262 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.18.96.193:44761)
0501 02:02:22.186395 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.186399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.186524 (+ 125us) server_negotiation.cc:408] Connection header received
0501 02:02:22.186636 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.186638 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.186676 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.186741 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:22.187491 (+ 750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.187959 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.188533 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.188706 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.190856 (+ 2150us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:22.190876 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:22.190885 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:22.190905 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:22.192336 (+ 1431us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.192734 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.192739 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.192741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.192782 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.192970 (+ 188us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.192972 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.192973 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.193117 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:22.193263 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.193419 (+ 156us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.193512 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:22.217216 19881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.209896 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.18.96.193:44761)
0501 02:02:22.210062 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.210066 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.210208 (+ 142us) server_negotiation.cc:408] Connection header received
0501 02:02:22.210356 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.210365 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.210403 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.210499 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:22.211240 (+ 741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.211687 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.212297 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.212503 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.214405 (+ 1902us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:22.214430 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:22.214433 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:22.214464 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:22.215779 (+ 1315us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.216197 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.216200 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.216202 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.216239 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.216464 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.216467 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.216469 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.216581 (+ 112us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:22.216716 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.216942 (+ 226us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.217035 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:22.238524 19881 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:22.231643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.18.96.193:44761)
0501 02:02:22.231802 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:22.231805 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:22.231868 (+ 63us) server_negotiation.cc:408] Connection header received
0501 02:02:22.231954 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:22.231956 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:22.231992 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:22.232055 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:22.232728 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.233157 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:22.233702 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:22.233814 (+ 112us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:22.236027 (+ 2213us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:22.236041 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:22.236043 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:22.236063 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:22.237392 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.237725 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.237728 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.237729 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.237764 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:22.237930 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:22.237931 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:22.237932 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:22.238089 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:22.238182 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:22.238339 (+ 157us) server_negotiation.cc:300] Negotiation successful
0501 02:02:22.238415 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260501 02:02:22.239748 19508 tablet_service.cc:3018] Rejecting scan request for tablet fc56502626ec4619ab4be5ad38a94137: Uninitialized: clean time has not yet been initialized
I20260501 02:02:22.241276 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19465
I20260501 02:02:22.247591 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19601
I20260501 02:02:22.252903 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19737
I20260501 02:02:22.258215 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19369
2026-05-01T02:02:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3122 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19908](info): set up 2 sockets
May 01 02:02:22 dist-test-slave-2j90 krb5kdc[19908](info): commencing operation
krb5kdc: starting...
W20260501 02:02:24.230336 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.950s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:24.360991 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40761
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:24.456813 19924 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:24.457062 19924 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:24.457149 19924 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:24.460479 19924 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:24.460592 19924 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260501 02:02:24.460629 19924 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:24.460673 19924 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:24.460713 19924 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:24.460752 19924 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:24.464892 19924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:24.465936 19924 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:24.466852 19924 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:24.472223 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.472223 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.472241 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:24.472362 19924 server_base.cc:1061] running on GCE node
I20260501 02:02:24.473063 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:24.473825 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:24.474982 19924 hybrid_clock.cc:648] HybridClock initialized: now 1777600944474957 us; error 39 us; skew 500 ppm
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:24.477408 19924 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:24.478502 19924 webserver.cc:492] Webserver started at http://127.18.96.254:36789/ using document root <none> and password file <none>
I20260501 02:02:24.479054 19924 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:24.479126 19924 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:24.479316 19924 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:24.480870 19924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "57a1be23c52a4e39a26c065e8fe5bcec"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "fae962bbc2ac17da1aec017ec03b11db"
server_key_iv: "8f07f55358de57e430bef383992713c9"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.481371 19924 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "57a1be23c52a4e39a26c065e8fe5bcec"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "fae962bbc2ac17da1aec017ec03b11db"
server_key_iv: "8f07f55358de57e430bef383992713c9"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.484444 19924 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:24.486387 19939 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:24.487237 19924 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:24.487363 19924 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "57a1be23c52a4e39a26c065e8fe5bcec"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "fae962bbc2ac17da1aec017ec03b11db"
server_key_iv: "8f07f55358de57e430bef383992713c9"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.487458 19924 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:24.498329 19924 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:24.500533 19924 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:24.500712 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:24.509037 19924 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40761
I20260501 02:02:24.509044 20001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40761 every 8 connection(s)
I20260501 02:02:24.509912 19924 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:24.512328 20002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:24.515519 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 19924
I20260501 02:02:24.515637 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:24.515861 18819 external_mini_cluster.cc:1468] Setting key d0c34891e8863df030c62b54ea113bf1
I20260501 02:02:24.517104 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Bootstrap starting.
I20260501 02:02:24.519071 20002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:24.519733 20002 log.cc:826] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:24.521423 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: No bootstrap required, opened a new log
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:24.524441 20002 raft_consensus.cc:359] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } }
I20260501 02:02:24.524674 20002 raft_consensus.cc:385] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:24.524750 20002 raft_consensus.cc:740] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a1be23c52a4e39a26c065e8fe5bcec, State: Initialized, Role: FOLLOWER
I20260501 02:02:24.525218 20002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } }
I20260501 02:02:24.525380 20002 raft_consensus.cc:399] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:24.525451 20002 raft_consensus.cc:493] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:24.525556 20002 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:24.526571 20002 raft_consensus.cc:515] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } }
I20260501 02:02:24.526942 20002 leader_election.cc:304] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a1be23c52a4e39a26c065e8fe5bcec; no voters:
I20260501 02:02:24.527261 20002 leader_election.cc:290] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:24.527390 20007 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:24.527611 20007 raft_consensus.cc:697] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [term 1 LEADER]: Becoming Leader. State: Replica: 57a1be23c52a4e39a26c065e8fe5bcec, State: Running, Role: LEADER
I20260501 02:02:24.527925 20007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } }
I20260501 02:02:24.528365 20002 sys_catalog.cc:565] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:24.529415 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.517003 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.18.96.254:40761)
0501 02:02:24.517394 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:24.517402 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:24.517430 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:02:24.518008 (+ 578us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:24.518034 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:24.518275 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:24.518567 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:24.519464 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.520399 (+ 935us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.521071 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.521380 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.523634 (+ 2254us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:24.523658 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:24.523672 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:24.523704 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:24.526425 (+ 2721us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.526903 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.526909 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.526914 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.526998 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.527236 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.527240 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.527242 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.527608 (+ 366us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:24.528314 (+ 706us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:24.528477 (+ 163us) server_negotiation.cc:300] Negotiation successful
0501 02:02:24.528703 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260501 02:02:24.529609 20009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a1be23c52a4e39a26c065e8fe5bcec. Latest consensus state: current_term: 1 leader_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } } }
I20260501 02:02:24.529762 20009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:24.530035 20008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a1be23c52a4e39a26c065e8fe5bcec" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40761 } } }
I20260501 02:02:24.530107 20008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:24.530181 20016 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:24.533102 20016 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:24.537794 20016 catalog_manager.cc:1357] Generated new cluster ID: 0c15bf2da28141c2b7fce37c2d3e9380
I20260501 02:02:24.537891 20016 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:24.559554 20016 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:24.560350 20016 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:24.567679 20016 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Generated new TSK 0
I20260501 02:02:24.568449 20016 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:24.622594 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40761
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:24.719317 20030 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:24.719534 20030 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:24.719625 20030 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:24.722833 20030 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:24.722920 20030 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:24.723023 20030 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:24.723114 20030 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:24.727195 20030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:24.728183 20030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:24.728971 20030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:24.735423 20035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.735482 20038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.735421 20036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:24.735515 20030 server_base.cc:1061] running on GCE node
I20260501 02:02:24.736016 20030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:24.736488 20030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:24.737628 20030 hybrid_clock.cc:648] HybridClock initialized: now 1777600944737600 us; error 41 us; skew 500 ppm
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:24.740221 20030 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:24.741199 20030 webserver.cc:492] Webserver started at http://127.18.96.193:36715/ using document root <none> and password file <none>
I20260501 02:02:24.741763 20030 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:24.741837 20030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:24.742091 20030 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:24.743680 20030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "ead76dc86134a64e1c1ec5c0f5ca5825"
server_key_iv: "f59acc60b7b1a2bafd7df6126f74874b"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.744166 20030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "ead76dc86134a64e1c1ec5c0f5ca5825"
server_key_iv: "f59acc60b7b1a2bafd7df6126f74874b"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.747331 20030 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260501 02:02:24.749300 20045 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:24.750276 20030 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:24.750398 20030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "ead76dc86134a64e1c1ec5c0f5ca5825"
server_key_iv: "f59acc60b7b1a2bafd7df6126f74874b"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.750495 20030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:24.764417 20030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:24.767230 20030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:24.767422 20030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:24.768860 20030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:24.768908 20030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:24.768975 20030 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:24.769017 20030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:24.780122 20030 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:37769
I20260501 02:02:24.780403 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.769361 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:43641 (local address 127.18.96.254:40761)
0501 02:02:24.769522 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:24.769527 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:24.770253 (+ 726us) server_negotiation.cc:408] Connection header received
0501 02:02:24.771078 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:24.771084 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:24.771147 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:24.771249 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:24.772532 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.773359 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.774218 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.774403 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.776841 (+ 2438us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:24.776878 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:24.776880 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:24.776905 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:24.778472 (+ 1567us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.778944 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.778948 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.778950 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.779006 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.779342 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.779346 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.779347 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.779493 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:24.779597 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:24.780119 (+ 522us) server_negotiation.cc:300] Negotiation successful
0501 02:02:24.780251 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260501 02:02:24.780964 20054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.769639 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.193:43641)
0501 02:02:24.770104 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:24.770140 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:24.770885 (+ 745us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:24.771440 (+ 555us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:24.771457 (+ 17us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:24.771847 (+ 390us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:24.772400 (+ 553us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.772412 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.773481 (+ 1069us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.773485 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:24.774101 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.774109 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.774300 (+ 191us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.774843 (+ 543us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:24.774874 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:24.776688 (+ 1814us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:24.778591 (+ 1903us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.778597 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.778607 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.778841 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.779109 (+ 268us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.779112 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.779113 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.779247 (+ 134us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.779598 (+ 351us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:24.779604 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:24.779831 (+ 227us) client_negotiation.cc:770] Sending connection context
0501 02:02:24.780051 (+ 220us) client_negotiation.cc:241] Negotiation successful
0501 02:02:24.780286 (+ 235us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260501 02:02:24.781113 20030 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:24.781641 20160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:37769 every 8 connection(s)
I20260501 02:02:24.782835 20161 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40761
I20260501 02:02:24.782990 20161 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:24.783521 20161 heartbeater.cc:507] Master 127.18.96.254:40761 requested a full tablet report, sending...
I20260501 02:02:24.785033 19956 ts_manager.cc:194] Registered new tserver with Master: 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769)
I20260501 02:02:24.786345 19956 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:43641
I20260501 02:02:24.787973 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20030
I20260501 02:02:24.788084 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:24.788322 18819 external_mini_cluster.cc:1468] Setting key c0fd47e24b1e8c643634efeadfe0720f
I20260501 02:02:24.790638 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.784362 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:33997 (local address 127.18.96.254:40761)
0501 02:02:24.784489 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:24.784492 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:24.784610 (+ 118us) server_negotiation.cc:408] Connection header received
0501 02:02:24.784725 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:24.784727 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:24.784765 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:24.784822 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:24.785434 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.785892 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.786686 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.786864 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.787672 (+ 808us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:24.787687 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:24.787690 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:24.787745 (+ 55us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:24.789293 (+ 1548us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.789662 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.789665 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.789667 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.789711 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.789946 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.789949 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.789950 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.790102 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:24.790199 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:24.790429 (+ 230us) server_negotiation.cc:300] Negotiation successful
0501 02:02:24.790517 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:02:24.790652 20167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.784245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.193:33997)
0501 02:02:24.784521 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:24.784535 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:24.784631 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:24.784861 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:24.784864 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:24.785047 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:24.785312 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.785319 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.786036 (+ 717us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.786039 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:24.786582 (+ 543us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.786590 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.786700 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.787291 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:24.787303 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:24.787537 (+ 234us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:24.789411 (+ 1874us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.789413 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.789415 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.789557 (+ 142us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.789807 (+ 250us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.789814 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.789815 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.789855 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.790201 (+ 346us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:24.790205 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:24.790280 (+ 75us) client_negotiation.cc:770] Sending connection context
0501 02:02:24.790380 (+ 100us) client_negotiation.cc:241] Negotiation successful
0501 02:02:24.790489 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:24.833709 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40761
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:24.931658 20172 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:24.931885 20172 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:24.931986 20172 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:24.935330 20172 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:24.935416 20172 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:24.935494 20172 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:24.935583 20172 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:24.939731 20172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:24.940699 20172 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:24.941495 20172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:24.947710 20178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.947710 20180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:24.947939 20177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:24.948118 20172 server_base.cc:1061] running on GCE node
I20260501 02:02:24.948498 20172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:24.949023 20172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:24.950196 20172 hybrid_clock.cc:648] HybridClock initialized: now 1777600944950184 us; error 32 us; skew 500 ppm
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:24.952684 20172 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:24.953667 20172 webserver.cc:492] Webserver started at http://127.18.96.194:33385/ using document root <none> and password file <none>
I20260501 02:02:24.954282 20172 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:24.954355 20172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:24.954561 20172 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:24.956303 20172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "5bd406cd16624132bc707efda1059ad3"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "f1c3651217596b0d442173451be73980"
server_key_iv: "9f3aedf9327aa982a07d38707216cf95"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.956768 20172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "5bd406cd16624132bc707efda1059ad3"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "f1c3651217596b0d442173451be73980"
server_key_iv: "9f3aedf9327aa982a07d38707216cf95"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.959967 20172 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260501 02:02:24.961935 20187 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:24.962906 20172 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260501 02:02:24.963035 20172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "5bd406cd16624132bc707efda1059ad3"
format_stamp: "Formatted at 2026-05-01 02:02:24 on dist-test-slave-2j90"
server_key: "f1c3651217596b0d442173451be73980"
server_key_iv: "9f3aedf9327aa982a07d38707216cf95"
server_key_version: "encryptionkey@0"
I20260501 02:02:24.963131 20172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:24.980453 20172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:24.982868 20172 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:24.983054 20172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:24.984539 20172 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:24.984606 20172 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:24.984701 20172 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:24.984730 20172 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:24 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:24.996165 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.985089 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55651 (local address 127.18.96.254:40761)
0501 02:02:24.985227 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:24.985232 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:24.985904 (+ 672us) server_negotiation.cc:408] Connection header received
0501 02:02:24.986887 (+ 983us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:24.986894 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:24.986954 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:24.987055 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:24.988209 (+ 1154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.988706 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.989335 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.989509 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.992246 (+ 2737us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:24.992265 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:24.992268 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:24.992301 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:24.994038 (+ 1737us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.994665 (+ 627us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.994669 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.994671 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.994725 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:24.995114 (+ 389us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:24.995117 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:24.995119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:24.995297 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:24.995384 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:24.995895 (+ 511us) server_negotiation.cc:300] Negotiation successful
0501 02:02:24.996039 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:02:24.996215 20172 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:36991
I20260501 02:02:24.996275 20302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:36991 every 8 connection(s)
I20260501 02:02:24.996811 20196 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:24.985371 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.194:55651)
0501 02:02:24.985749 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:24.985792 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:24.986680 (+ 888us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:24.987175 (+ 495us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:24.987183 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:24.987560 (+ 377us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:24.988068 (+ 508us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.988080 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.988830 (+ 750us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:24.988836 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:24.989220 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:24.989226 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:24.989411 (+ 185us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:24.989992 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:24.990029 (+ 37us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:24.992077 (+ 2048us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:24.994184 (+ 2107us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.994193 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.994208 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.994549 (+ 341us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.994833 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:24.994838 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:24.994842 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:24.995007 (+ 165us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:24.995410 (+ 403us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:24.995415 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:24.995631 (+ 216us) client_negotiation.cc:770] Sending connection context
0501 02:02:24.995842 (+ 211us) client_negotiation.cc:241] Negotiation successful
0501 02:02:24.996088 (+ 246us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260501 02:02:24.997232 20172 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:24.998693 20303 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40761
I20260501 02:02:24.998850 20303 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:24.999100 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20172
I20260501 02:02:24.999235 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:24.999387 20303 heartbeater.cc:507] Master 127.18.96.254:40761 requested a full tablet report, sending...
I20260501 02:02:24.999486 18819 external_mini_cluster.cc:1468] Setting key dbe94f383d7341276e0b596f31cd13aa
I20260501 02:02:25.000329 19956 ts_manager.cc:194] Registered new tserver with Master: 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991)
I20260501 02:02:25.000985 19956 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:55651
I20260501 02:02:25.007565 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.000223 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:60999 (local address 127.18.96.254:40761)
0501 02:02:25.000769 (+ 546us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.000773 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.000786 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:02:25.000814 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.000817 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.000852 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.000927 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:25.002694 (+ 1767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.003161 (+ 467us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.003773 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.003981 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.004688 (+ 707us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:25.004704 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:25.004707 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:25.004736 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:25.006174 (+ 1438us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.006548 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.006553 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.006555 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.006601 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.006846 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.006850 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.006852 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.006992 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:25.007097 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.007340 (+ 243us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.007436 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260501 02:02:25.007581 20309 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.000164 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.194:60999)
0501 02:02:25.000423 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.000438 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.000519 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.002068 (+ 1549us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.002074 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.002278 (+ 204us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:25.002563 (+ 285us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.002570 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.003269 (+ 699us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.003275 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.003653 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.003659 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.003784 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.004310 (+ 526us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:25.004322 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:25.004569 (+ 247us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:25.006280 (+ 1711us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.006284 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.006287 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.006452 (+ 165us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.006697 (+ 245us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.006705 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.006706 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.006748 (+ 42us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.007088 (+ 340us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:25.007091 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:25.007170 (+ 79us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.007290 (+ 120us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.007407 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:25.044814 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40761
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:25.142334 20314 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:25.142517 20314 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:25.142632 20314 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:25.145810 20314 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:25.145867 20314 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:25.145920 20314 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:25.146037 20314 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:25.150290 20314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:25.151248 20314 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:25.152022 20314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:25.158041 20322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:25.158063 20319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:25.158062 20320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:25.158215 20314 server_base.cc:1061] running on GCE node
I20260501 02:02:25.158576 20314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:25.159139 20314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:25.160329 20314 hybrid_clock.cc:648] HybridClock initialized: now 1777600945160313 us; error 32 us; skew 500 ppm
May 01 02:02:25 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600945, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:25.162801 20314 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:25.163725 20314 webserver.cc:492] Webserver started at http://127.18.96.195:34659/ using document root <none> and password file <none>
I20260501 02:02:25.164266 20314 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:25.164340 20314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:25.164532 20314 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:25.166116 20314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "93f7c1b23fe64870a63b86600646ceff"
format_stamp: "Formatted at 2026-05-01 02:02:25 on dist-test-slave-2j90"
server_key: "11ca22c222c2f209cbb629e19925e9ca"
server_key_iv: "e474b3d0b7447ac2e515758ab25a36af"
server_key_version: "encryptionkey@0"
I20260501 02:02:25.166576 20314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "93f7c1b23fe64870a63b86600646ceff"
format_stamp: "Formatted at 2026-05-01 02:02:25 on dist-test-slave-2j90"
server_key: "11ca22c222c2f209cbb629e19925e9ca"
server_key_iv: "e474b3d0b7447ac2e515758ab25a36af"
server_key_version: "encryptionkey@0"
I20260501 02:02:25.169632 20314 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:02:25.171576 20329 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:25.172410 20314 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:25.172534 20314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "93f7c1b23fe64870a63b86600646ceff"
format_stamp: "Formatted at 2026-05-01 02:02:25 on dist-test-slave-2j90"
server_key: "11ca22c222c2f209cbb629e19925e9ca"
server_key_iv: "e474b3d0b7447ac2e515758ab25a36af"
server_key_version: "encryptionkey@0"
I20260501 02:02:25.172626 20314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:25.182789 20314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:25.184880 20314 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:25.185065 20314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:25.186359 20314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:25.186425 20314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:25.186496 20314 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:25.186525 20314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:25 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600945, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:25.197343 20314 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:44955
I20260501 02:02:25.197669 20444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:44955 every 8 connection(s)
I20260501 02:02:25.198343 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.186949 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:46257 (local address 127.18.96.254:40761)
0501 02:02:25.187098 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.187106 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.187683 (+ 577us) server_negotiation.cc:408] Connection header received
0501 02:02:25.188504 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.188509 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.188571 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.188679 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:25.190126 (+ 1447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.190824 (+ 698us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.191808 (+ 984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.191982 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.194679 (+ 2697us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:25.194699 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:25.194702 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:25.194733 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:25.196451 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.196872 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.196876 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.196878 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.196940 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.197334 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.197337 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.197338 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.197495 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:25.197583 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.198142 (+ 559us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.198228 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:02:25.198411 20314 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:25.198984 20338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.187167 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.195:46257)
0501 02:02:25.187540 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.187574 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.188311 (+ 737us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.188934 (+ 623us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.188946 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.189403 (+ 457us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:25.189935 (+ 532us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.189949 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.190978 (+ 1029us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.190981 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.191694 (+ 713us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.191705 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.191871 (+ 166us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.192512 (+ 641us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:25.192535 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:25.194540 (+ 2005us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:25.196543 (+ 2003us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.196548 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.196557 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.196757 (+ 200us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.197083 (+ 326us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.197089 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.197094 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.197211 (+ 117us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.197605 (+ 394us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:25.197611 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:25.197852 (+ 241us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.198084 (+ 232us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.198290 (+ 206us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
I20260501 02:02:25.199644 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20314
I20260501 02:02:25.199785 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:25.200027 18819 external_mini_cluster.cc:1468] Setting key 3be008e808e8d823e19c03cbb30fc3e0
I20260501 02:02:25.200614 20445 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40761
I20260501 02:02:25.200798 20445 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:25.201339 20445 heartbeater.cc:507] Master 127.18.96.254:40761 requested a full tablet report, sending...
I20260501 02:02:25.202296 19955 ts_manager.cc:194] Registered new tserver with Master: 93f7c1b23fe64870a63b86600646ceff (127.18.96.195:44955)
I20260501 02:02:25.203001 19955 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:46257
I20260501 02:02:25.208207 20451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.202686 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.18.96.195:47329)
0501 02:02:25.202985 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.203000 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.203105 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.203373 (+ 268us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.203376 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.203555 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:25.203773 (+ 218us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.203779 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.204655 (+ 876us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.204658 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.205011 (+ 353us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.205017 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.205099 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.205496 (+ 397us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:25.205508 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:25.205741 (+ 233us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:25.207218 (+ 1477us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.207221 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.207222 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.207377 (+ 155us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.207562 (+ 185us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.207564 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.207565 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.207601 (+ 36us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.207877 (+ 276us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:25.207879 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:25.207936 (+ 57us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.208015 (+ 79us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.208095 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":139,"threads_started":1}
I20260501 02:02:25.208233 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.202813 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:47329 (local address 127.18.96.254:40761)
0501 02:02:25.202929 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.202933 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.203102 (+ 169us) server_negotiation.cc:408] Connection header received
0501 02:02:25.203201 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.203205 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.203274 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.203346 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:25.203890 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.204553 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.205100 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.205216 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.205857 (+ 641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:25.205869 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:25.205871 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:25.205890 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:25.207135 (+ 1245us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.207455 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.207457 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.207458 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.207492 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.207667 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.207669 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.207670 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.207788 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:25.207877 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.208066 (+ 189us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.208138 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260501 02:02:25.213626 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
May 01 02:02:25 dist-test-slave-2j90 krb5kdc[19908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:02:25 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:25.234779 20005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.228314 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.18.96.254:40761)
0501 02:02:25.228435 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.228438 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.228450 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:02:25.228482 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.228484 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.228522 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.228589 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:25.229306 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.229798 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.230418 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.230588 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.232508 (+ 1920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:25.232522 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:25.232524 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:25.232544 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:25.233670 (+ 1126us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.234063 (+ 393us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.234066 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.234067 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.234104 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.234270 (+ 166us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.234271 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.234272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.234377 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:25.234465 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.234598 (+ 133us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.234672 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:02:25.237200 19955 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:25.239506 19955 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:25.251178 20462 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.246945 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:44955 (local address 127.0.0.1:51276)
0501 02:02:25.247263 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.247314 (+ 51us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.247450 (+ 136us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.248193 (+ 743us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.248198 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.248219 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.248508 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.248514 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.249696 (+ 1182us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.249699 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.250696 (+ 997us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.250705 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.250811 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.250866 (+ 55us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.250950 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.251048 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260501 02:02:25.251837 20466 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.247990 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:36991 (local address 127.0.0.1:40514)
0501 02:02:25.248331 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.248344 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.248440 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.249186 (+ 746us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.249189 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.249203 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.249445 (+ 242us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.249450 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.250572 (+ 1122us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.250576 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.251467 (+ 891us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.251477 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.251581 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.251593 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.251638 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.251688 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"spinlock_wait_cycles":17408,"thread_start_us":95,"threads_started":1}
I20260501 02:02:25.251961 20463 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.247125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.18.96.195:44955)
0501 02:02:25.247716 (+ 591us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.247722 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.247743 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:02:25.247821 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.247827 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.248003 (+ 176us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.248623 (+ 620us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.248739 (+ 116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.249582 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.251076 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.251558 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.251661 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.251734 (+ 73us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.251808 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":91,"threads_started":1}
I20260501 02:02:25.252760 20465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.247591 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.18.96.193:37769)
0501 02:02:25.247965 (+ 374us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.247971 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.248551 (+ 580us) server_negotiation.cc:408] Connection header received
0501 02:02:25.248639 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.248643 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.248802 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.248937 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.249602 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.250787 (+ 1185us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.251897 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.252416 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.252497 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.252556 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.252632 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":95,"threads_started":1}
I20260501 02:02:25.253041 20467 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.248324 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.18.96.194:36991)
0501 02:02:25.248683 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.248688 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.248704 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:02:25.248787 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.248791 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.248925 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.249054 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.249573 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.250430 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.252145 (+ 1715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.252497 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.252637 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.252696 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.252913 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
I20260501 02:02:25.254822 20378 tablet_service.cc:1511] Processing CreateTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e (DEFAULT_TABLE table=test-table [id=4b3f96fb841544c8b738270039cd0b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.255316 20237 tablet_service.cc:1511] Processing CreateTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e (DEFAULT_TABLE table=test-table [id=4b3f96fb841544c8b738270039cd0b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.255713 20378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc9883202ba4ad39889b97f89f19d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.256099 20237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc9883202ba4ad39889b97f89f19d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.260638 20469 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Bootstrap starting.
I20260501 02:02:25.261061 20464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.247176 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:37769 (local address 127.0.0.1:46722)
0501 02:02:25.248320 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.248333 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.248448 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.249127 (+ 679us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.249130 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.249148 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.249443 (+ 295us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.249450 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.250900 (+ 1450us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.250903 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.251754 (+ 851us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.251769 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.251894 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.251907 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.251956 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.252006 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"spinlock_wait_cycles":2304,"thread_start_us":96,"threads_started":1}
I20260501 02:02:25.261438 20468 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Bootstrap starting.
I20260501 02:02:25.263006 20469 tablet_bootstrap.cc:654] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.263567 20468 tablet_bootstrap.cc:654] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.264030 20469 log.cc:826] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:25.263897 20095 tablet_service.cc:1511] Processing CreateTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e (DEFAULT_TABLE table=test-table [id=4b3f96fb841544c8b738270039cd0b00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.264671 20468 log.cc:826] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:25.264776 20095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fc9883202ba4ad39889b97f89f19d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.265637 20469 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: No bootstrap required, opened a new log
I20260501 02:02:25.265828 20469 ts_tablet_manager.cc:1403] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:02:25.266379 20468 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: No bootstrap required, opened a new log
I20260501 02:02:25.266541 20468 ts_tablet_manager.cc:1403] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:02:25.268445 20469 raft_consensus.cc:359] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.268651 20469 raft_consensus.cc:385] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.268724 20469 raft_consensus.cc:740] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7c1b23fe64870a63b86600646ceff, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.269132 20469 consensus_queue.cc:260] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.269711 20445 heartbeater.cc:499] Master 127.18.96.254:40761 was elected leader, sending a full tablet report...
I20260501 02:02:25.269652 20468 raft_consensus.cc:359] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.269836 20468 raft_consensus.cc:385] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.269881 20468 raft_consensus.cc:740] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd406cd16624132bc707efda1059ad3, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.269975 20469 ts_tablet_manager.cc:1434] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:02:25.270262 20468 consensus_queue.cc:260] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.271098 20468 ts_tablet_manager.cc:1434] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:02:25.271212 20303 heartbeater.cc:499] Master 127.18.96.254:40761 was elected leader, sending a full tablet report...
I20260501 02:02:25.271519 20473 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Bootstrap starting.
I20260501 02:02:25.273636 20473 tablet_bootstrap.cc:654] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.274858 20473 log.cc:826] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:25.276372 20473 tablet_bootstrap.cc:492] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: No bootstrap required, opened a new log
I20260501 02:02:25.276546 20473 ts_tablet_manager.cc:1403] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:02:25.278712 20473 raft_consensus.cc:359] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.278869 20473 raft_consensus.cc:385] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.278935 20473 raft_consensus.cc:740] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bee1dd6b9ff4a36be739fa3147fd517, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.279310 20473 consensus_queue.cc:260] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.279867 20161 heartbeater.cc:499] Master 127.18.96.254:40761 was elected leader, sending a full tablet report...
I20260501 02:02:25.280055 20473 ts_tablet_manager.cc:1434] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260501 02:02:25.282533 20162 tablet.cc:2404] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:25.388006 20474 raft_consensus.cc:493] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:25.388219 20474 raft_consensus.cc:515] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.389334 20474 leader_election.cc:290] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f7c1b23fe64870a63b86600646ceff (127.18.96.195:44955), 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769)
I20260501 02:02:25.392771 20196 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.389579 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:37769 (local address 127.18.96.194:40283)
0501 02:02:25.389715 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.389732 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.389884 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.390296 (+ 412us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.390300 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.390321 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.390591 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.390603 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.391515 (+ 912us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.391518 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.392359 (+ 841us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.392368 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.392485 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.392502 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.392552 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.392602 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260501 02:02:25.393078 20465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.389682 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40283 (local address 127.18.96.193:37769)
0501 02:02:25.389851 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.389856 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.389875 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:02:25.390043 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.390047 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.390102 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.390185 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.390737 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.391423 (+ 686us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.392483 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.392817 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.392881 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.392937 (+ 56us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.392987 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:02:25.393136 20477 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.389579 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:44955 (local address 127.18.96.194:37679)
0501 02:02:25.390340 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.390353 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.390454 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.390691 (+ 237us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.390694 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.390715 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.390939 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.390946 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.391883 (+ 937us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.391886 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.392763 (+ 877us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.392771 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.392879 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.392898 (+ 19us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.392948 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.393003 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":231,"thread_start_us":170,"threads_started":1}
I20260501 02:02:25.393765 20463 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.389755 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:37679 (local address 127.18.96.195:44955)
0501 02:02:25.389907 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.389913 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.390444 (+ 531us) server_negotiation.cc:408] Connection header received
0501 02:02:25.390549 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.390552 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.390591 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.390652 (+ 61us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.391071 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.391774 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.393082 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.393535 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.393566 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.393614 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.393672 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260501 02:02:25.393885 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fc9883202ba4ad39889b97f89f19d9e" candidate_uuid: "5bd406cd16624132bc707efda1059ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" is_pre_election: true
I20260501 02:02:25.394169 20100 raft_consensus.cc:2468] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd406cd16624132bc707efda1059ad3 in term 0.
I20260501 02:02:25.394307 20399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fc9883202ba4ad39889b97f89f19d9e" candidate_uuid: "5bd406cd16624132bc707efda1059ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7c1b23fe64870a63b86600646ceff" is_pre_election: true
I20260501 02:02:25.394534 20399 raft_consensus.cc:2468] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bd406cd16624132bc707efda1059ad3 in term 0.
I20260501 02:02:25.394652 20191 leader_election.cc:304] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd406cd16624132bc707efda1059ad3, 7bee1dd6b9ff4a36be739fa3147fd517; no voters:
I20260501 02:02:25.394932 20474 raft_consensus.cc:2804] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:25.395008 20474 raft_consensus.cc:493] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:25.395058 20474 raft_consensus.cc:3060] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.395967 20474 raft_consensus.cc:515] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.396296 20474 leader_election.cc:290] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f7c1b23fe64870a63b86600646ceff (127.18.96.195:44955), 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769)
I20260501 02:02:25.396577 20399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fc9883202ba4ad39889b97f89f19d9e" candidate_uuid: "5bd406cd16624132bc707efda1059ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f7c1b23fe64870a63b86600646ceff"
I20260501 02:02:25.396692 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fc9883202ba4ad39889b97f89f19d9e" candidate_uuid: "5bd406cd16624132bc707efda1059ad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
I20260501 02:02:25.396760 20399 raft_consensus.cc:3060] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.396852 20100 raft_consensus.cc:3060] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.397686 20399 raft_consensus.cc:2468] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd406cd16624132bc707efda1059ad3 in term 1.
I20260501 02:02:25.397789 20100 raft_consensus.cc:2468] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bd406cd16624132bc707efda1059ad3 in term 1.
I20260501 02:02:25.398029 20188 leader_election.cc:304] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd406cd16624132bc707efda1059ad3, 93f7c1b23fe64870a63b86600646ceff; no voters:
I20260501 02:02:25.398205 20474 raft_consensus.cc:2804] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:25.398409 20474 raft_consensus.cc:697] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd406cd16624132bc707efda1059ad3, State: Running, Role: LEADER
I20260501 02:02:25.398758 20474 consensus_queue.cc:237] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } }
I20260501 02:02:25.401556 19955 catalog_manager.cc:5671] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bd406cd16624132bc707efda1059ad3 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "5bd406cd16624132bc707efda1059ad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } health_report { overall_health: HEALTHY } } }
I20260501 02:02:25.441779 20467 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.438853 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.18.96.194:36991)
0501 02:02:25.439025 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.439028 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.439044 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:02:25.439214 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.439217 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.439272 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.439357 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:25.439664 (+ 307us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.440217 (+ 553us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.440853 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.441000 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.441052 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:25.441370 (+ 318us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:25.441455 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.441539 (+ 84us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.441581 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:02:25.447158 20399 raft_consensus.cc:1275] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Refusing update from remote peer 5bd406cd16624132bc707efda1059ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:25.447158 20100 raft_consensus.cc:1275] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Refusing update from remote peer 5bd406cd16624132bc707efda1059ad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:25.447820 20474 consensus_queue.cc:1048] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:25.448060 20478 consensus_queue.cc:1048] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 02:02:25.449206 20446 tablet.cc:2404] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:25.456308 20483 mvcc.cc:204] Tried to move back new op lower bound from 7281053472544718848 to 7281053472355721216. Current Snapshot: MvccSnapshot[applied={T|T < 7281053472544718848}]
I20260501 02:02:25.463260 19956 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:25.463418 19956 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:25.465827 19956 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Sending DeleteTablet for 3 replicas of tablet 5fc9883202ba4ad39889b97f89f19d9e
I20260501 02:02:25.466511 20378 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:25 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:51276
I20260501 02:02:25.466512 20095 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:25 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:46722
I20260501 02:02:25.466718 20237 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fc9883202ba4ad39889b97f89f19d9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:25 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:40514
I20260501 02:02:25.467044 20490 tablet_replica.cc:333] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: stopping tablet replica
I20260501 02:02:25.467334 20490 raft_consensus.cc:2243] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:25.467657 20490 raft_consensus.cc:2272] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:25.467993 19956 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:25.468425 19956 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:25.468966 20490 ts_tablet_manager.cc:1916] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:25.471222 20490 ts_tablet_manager.cc:1929] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:25.471320 20490 log.cc:1199] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/5fc9883202ba4ad39889b97f89f19d9e
I20260501 02:02:25.471617 20490 ts_tablet_manager.cc:1950] T 5fc9883202ba4ad39889b97f89f19d9e P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting consensus metadata
I20260501 02:02:25.472558 20491 tablet_replica.cc:333] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: stopping tablet replica
I20260501 02:02:25.468060 20492 tablet_replica.cc:333] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: stopping tablet replica
I20260501 02:02:25.472811 20491 raft_consensus.cc:2243] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:25.472810 19943 catalog_manager.cc:5002] TS 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769): tablet 5fc9883202ba4ad39889b97f89f19d9e (table test-table [id=4b3f96fb841544c8b738270039cd0b00]) successfully deleted
I20260501 02:02:25.473006 20492 raft_consensus.cc:2243] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:25.473083 20491 raft_consensus.cc:2272] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:25.473341 20492 raft_consensus.cc:2272] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:25.473453 20095 tablet_service.cc:1511] Processing CreateTablet for tablet 348c6a9af78944baaacff8c6697dbb0c (DEFAULT_TABLE table=test-table [id=7302df9894c24e2b9498447a7d697b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.473685 20095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c6a9af78944baaacff8c6697dbb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.473697 20237 tablet_service.cc:1511] Processing CreateTablet for tablet 348c6a9af78944baaacff8c6697dbb0c (DEFAULT_TABLE table=test-table [id=7302df9894c24e2b9498447a7d697b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.473894 20237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c6a9af78944baaacff8c6697dbb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.475549 20468 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Bootstrap starting.
I20260501 02:02:25.476621 20468 tablet_bootstrap.cc:654] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.477249 20473 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Bootstrap starting.
I20260501 02:02:25.477882 20468 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: No bootstrap required, opened a new log
I20260501 02:02:25.477996 20468 ts_tablet_manager.cc:1403] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:25.478247 20473 tablet_bootstrap.cc:654] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.478487 20468 raft_consensus.cc:359] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.478588 20468 raft_consensus.cc:385] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.478621 20468 raft_consensus.cc:740] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd406cd16624132bc707efda1059ad3, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.478886 20468 consensus_queue.cc:260] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.478962 20378 tablet_service.cc:1511] Processing CreateTablet for tablet 348c6a9af78944baaacff8c6697dbb0c (DEFAULT_TABLE table=test-table [id=7302df9894c24e2b9498447a7d697b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:25.479200 20378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 348c6a9af78944baaacff8c6697dbb0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.479735 20468 ts_tablet_manager.cc:1434] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:25.481280 20469 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Bootstrap starting.
I20260501 02:02:25.482295 20469 tablet_bootstrap.cc:654] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.482668 20492 ts_tablet_manager.cc:1916] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:25.483837 20469 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: No bootstrap required, opened a new log
I20260501 02:02:25.483928 20469 ts_tablet_manager.cc:1403] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:25.484417 20469 raft_consensus.cc:359] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.484498 20469 raft_consensus.cc:385] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.484524 20469 raft_consensus.cc:740] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7c1b23fe64870a63b86600646ceff, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.484656 20469 consensus_queue.cc:260] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.484956 20469 ts_tablet_manager.cc:1434] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:02:25.485054 20492 ts_tablet_manager.cc:1929] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:25.485142 20492 log.cc:1199] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/5fc9883202ba4ad39889b97f89f19d9e
I20260501 02:02:25.485440 20492 ts_tablet_manager.cc:1950] T 5fc9883202ba4ad39889b97f89f19d9e P 5bd406cd16624132bc707efda1059ad3: Deleting consensus metadata
I20260501 02:02:25.486670 19942 catalog_manager.cc:5002] TS 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991): tablet 5fc9883202ba4ad39889b97f89f19d9e (table test-table [id=4b3f96fb841544c8b738270039cd0b00]) successfully deleted
I20260501 02:02:25.486726 20473 tablet_bootstrap.cc:492] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: No bootstrap required, opened a new log
I20260501 02:02:25.486795 20473 ts_tablet_manager.cc:1403] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260501 02:02:25.486816 20491 ts_tablet_manager.cc:1916] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:25.487264 20473 raft_consensus.cc:359] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.487370 20473 raft_consensus.cc:385] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.487409 20473 raft_consensus.cc:740] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bee1dd6b9ff4a36be739fa3147fd517, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.487535 20473 consensus_queue.cc:260] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.487772 20473 ts_tablet_manager.cc:1434] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:25.489256 20491 ts_tablet_manager.cc:1929] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:25.489341 20491 log.cc:1199] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/5fc9883202ba4ad39889b97f89f19d9e
I20260501 02:02:25.489686 20491 ts_tablet_manager.cc:1950] T 5fc9883202ba4ad39889b97f89f19d9e P 93f7c1b23fe64870a63b86600646ceff: Deleting consensus metadata
I20260501 02:02:25.490671 19940 catalog_manager.cc:5002] TS 93f7c1b23fe64870a63b86600646ceff (127.18.96.195:44955): tablet 5fc9883202ba4ad39889b97f89f19d9e (table test-table [id=4b3f96fb841544c8b738270039cd0b00]) successfully deleted
W20260501 02:02:25.498389 20304 tablet.cc:2404] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:25.888751 20472 raft_consensus.cc:493] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:25.888903 20472 raft_consensus.cc:515] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.889953 20472 leader_election.cc:290] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769), 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991)
I20260501 02:02:25.893072 20497 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.890212 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:37769 (local address 127.18.96.195:32771)
0501 02:02:25.890551 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.890564 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.890697 (+ 133us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.890981 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.890985 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.891002 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.891237 (+ 235us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.891242 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.892008 (+ 766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.892014 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.892583 (+ 569us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.892589 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.892716 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.892733 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.892817 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.892885 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":72,"threads_started":1}
I20260501 02:02:25.893203 20496 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.890250 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:36991 (local address 127.18.96.195:35145)
0501 02:02:25.890667 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.890678 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.890792 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.891080 (+ 288us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.891084 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.891101 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:25.891310 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.891316 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.892130 (+ 814us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.892135 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.892690 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.892696 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.892867 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.892882 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.892949 (+ 67us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.893001 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":89,"threads_started":1}
I20260501 02:02:25.893536 20465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.890280 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:32771 (local address 127.18.96.193:37769)
0501 02:02:25.890436 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.890441 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.890651 (+ 210us) server_negotiation.cc:408] Connection header received
0501 02:02:25.890813 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.890816 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.890871 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.890956 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.891349 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.891884 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.892701 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.893260 (+ 559us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.893299 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.893350 (+ 51us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.893403 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:02:25.893549 20467 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.890344 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:35145 (local address 127.18.96.194:36991)
0501 02:02:25.890603 (+ 259us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.890607 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.890746 (+ 139us) server_negotiation.cc:408] Connection header received
0501 02:02:25.890908 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.890911 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.890966 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.891071 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:25.891420 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.892014 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.892839 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.893354 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.893382 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.893430 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.893468 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260501 02:02:25.894017 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "348c6a9af78944baaacff8c6697dbb0c" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" is_pre_election: true
I20260501 02:02:25.894162 20239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "348c6a9af78944baaacff8c6697dbb0c" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd406cd16624132bc707efda1059ad3" is_pre_election: true
I20260501 02:02:25.894210 20100 raft_consensus.cc:2468] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 0.
I20260501 02:02:25.894400 20239 raft_consensus.cc:2468] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 0.
I20260501 02:02:25.894618 20333 leader_election.cc:304] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bee1dd6b9ff4a36be739fa3147fd517, 93f7c1b23fe64870a63b86600646ceff; no voters:
I20260501 02:02:25.894862 20472 raft_consensus.cc:2804] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:25.894924 20472 raft_consensus.cc:493] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:25.894953 20472 raft_consensus.cc:3060] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.895810 20472 raft_consensus.cc:515] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.896224 20472 leader_election.cc:290] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 election: Requested vote from peers 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769), 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991)
I20260501 02:02:25.896534 20239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "348c6a9af78944baaacff8c6697dbb0c" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd406cd16624132bc707efda1059ad3"
I20260501 02:02:25.896534 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "348c6a9af78944baaacff8c6697dbb0c" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
I20260501 02:02:25.896646 20239 raft_consensus.cc:3060] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.896646 20100 raft_consensus.cc:3060] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.897655 20100 raft_consensus.cc:2468] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 1.
I20260501 02:02:25.897734 20239 raft_consensus.cc:2468] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 1.
I20260501 02:02:25.898010 20333 leader_election.cc:304] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bee1dd6b9ff4a36be739fa3147fd517, 93f7c1b23fe64870a63b86600646ceff; no voters:
I20260501 02:02:25.898198 20472 raft_consensus.cc:2804] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:25.898341 20472 raft_consensus.cc:697] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 1 LEADER]: Becoming Leader. State: Replica: 93f7c1b23fe64870a63b86600646ceff, State: Running, Role: LEADER
I20260501 02:02:25.898559 20472 consensus_queue.cc:237] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.900637 19956 catalog_manager.cc:5671] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7c1b23fe64870a63b86600646ceff (127.18.96.195). New cstate: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } health_report { overall_health: HEALTHY } } }
May 01 02:02:25 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600944, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:25.958896 20505 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.951658 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40761 (local address 127.0.0.1:58946)
0501 02:02:25.951954 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.951967 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.952076 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.952304 (+ 228us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.952306 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.952691 (+ 385us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:25.952877 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.952882 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.953541 (+ 659us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.953544 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.953937 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.953946 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.954114 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.954668 (+ 554us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:25.954680 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:25.956024 (+ 1344us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:25.957515 (+ 1491us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.957524 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.957528 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.957754 (+ 226us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.958061 (+ 307us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:25.958068 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:25.958070 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:25.958111 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:25.958427 (+ 316us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:25.958433 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:25.958544 (+ 111us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.958667 (+ 123us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.958771 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260501 02:02:25.959067 20506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.951697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.18.96.254:40761)
0501 02:02:25.951993 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.951997 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.952012 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:02:25.952157 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.952159 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.952197 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.952272 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:25.952972 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.953454 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.954110 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.954262 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.956162 (+ 1900us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:25.956181 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:25.956184 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:25.956207 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:25.957408 (+ 1201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.957873 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.957884 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.957885 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.957923 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:25.958197 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:25.958201 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:25.958203 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:25.958326 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:25.958426 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.958733 (+ 307us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.958842 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":137,"threads_started":1}
I20260501 02:02:25.962441 19956 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:58946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260501 02:02:25.963182 19956 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:25.967871 20095 tablet_service.cc:1511] Processing CreateTablet for tablet 4ade33924972441e82a9c4b531512f55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e0bbdcfc0cb4bc8b336721c12b78913]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:25.967913 20237 tablet_service.cc:1511] Processing CreateTablet for tablet 4ade33924972441e82a9c4b531512f55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e0bbdcfc0cb4bc8b336721c12b78913]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:25.968108 20378 tablet_service.cc:1511] Processing CreateTablet for tablet 4ade33924972441e82a9c4b531512f55 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e0bbdcfc0cb4bc8b336721c12b78913]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:25.968245 20095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ade33924972441e82a9c4b531512f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.968266 20237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ade33924972441e82a9c4b531512f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.968402 20378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ade33924972441e82a9c4b531512f55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:25.970520 20473 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: Bootstrap starting.
I20260501 02:02:25.970736 20468 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: Bootstrap starting.
I20260501 02:02:25.970757 20469 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: Bootstrap starting.
I20260501 02:02:25.971452 20473 tablet_bootstrap.cc:654] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.971634 20468 tablet_bootstrap.cc:654] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.971634 20469 tablet_bootstrap.cc:654] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:25.972512 20473 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: No bootstrap required, opened a new log
I20260501 02:02:25.972620 20473 ts_tablet_manager.cc:1403] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:25.973035 20469 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: No bootstrap required, opened a new log
I20260501 02:02:25.973124 20469 ts_tablet_manager.cc:1403] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:25.973135 20473 raft_consensus.cc:359] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.973250 20473 raft_consensus.cc:385] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.973296 20473 raft_consensus.cc:740] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bee1dd6b9ff4a36be739fa3147fd517, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.973423 20473 consensus_queue.cc:260] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.973591 20469 raft_consensus.cc:359] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.973690 20469 raft_consensus.cc:385] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.973698 20473 ts_tablet_manager.cc:1434] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:02:25.973732 20469 raft_consensus.cc:740] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f7c1b23fe64870a63b86600646ceff, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.973778 20476 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:25.973887 20476 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:25.973868 20469 consensus_queue.cc:260] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.974205 20469 ts_tablet_manager.cc:1434] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:25.974166 20472 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:25.974275 20472 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:25.974462 20468 tablet_bootstrap.cc:492] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: No bootstrap required, opened a new log
I20260501 02:02:25.974541 20468 ts_tablet_manager.cc:1403] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260501 02:02:25.974951 20468 raft_consensus.cc:359] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.975077 20468 raft_consensus.cc:385] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:25.975152 20468 raft_consensus.cc:740] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd406cd16624132bc707efda1059ad3, State: Initialized, Role: FOLLOWER
I20260501 02:02:25.975270 20468 consensus_queue.cc:260] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.975512 20478 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:25.975636 20478 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:25.975524 20468 ts_tablet_manager.cc:1434] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:25.978682 20472 raft_consensus.cc:493] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:25.978750 20472 raft_consensus.cc:515] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.979050 20472 leader_election.cc:290] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769), 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991)
I20260501 02:02:25.979321 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ade33924972441e82a9c4b531512f55" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" is_pre_election: true
I20260501 02:02:25.979354 20239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ade33924972441e82a9c4b531512f55" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd406cd16624132bc707efda1059ad3" is_pre_election: true
I20260501 02:02:25.979434 20239 raft_consensus.cc:2468] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 0.
I20260501 02:02:25.979439 20100 raft_consensus.cc:2468] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 0.
I20260501 02:02:25.979704 20332 leader_election.cc:304] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bd406cd16624132bc707efda1059ad3, 93f7c1b23fe64870a63b86600646ceff; no voters:
I20260501 02:02:25.979872 20472 raft_consensus.cc:2804] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:25.979924 20472 raft_consensus.cc:493] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:25.979950 20472 raft_consensus.cc:3060] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.980767 20472 raft_consensus.cc:515] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.981137 20472 leader_election.cc:290] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 election: Requested vote from peers 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769), 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991)
I20260501 02:02:25.981416 20100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ade33924972441e82a9c4b531512f55" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bee1dd6b9ff4a36be739fa3147fd517"
I20260501 02:02:25.981436 20239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ade33924972441e82a9c4b531512f55" candidate_uuid: "93f7c1b23fe64870a63b86600646ceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bd406cd16624132bc707efda1059ad3"
I20260501 02:02:25.981501 20100 raft_consensus.cc:3060] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.981545 20239 raft_consensus.cc:3060] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:25.982365 20100 raft_consensus.cc:2468] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 1.
I20260501 02:02:25.982539 20239 raft_consensus.cc:2468] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93f7c1b23fe64870a63b86600646ceff in term 1.
I20260501 02:02:25.982651 20333 leader_election.cc:304] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bee1dd6b9ff4a36be739fa3147fd517, 93f7c1b23fe64870a63b86600646ceff; no voters:
I20260501 02:02:25.982843 20472 raft_consensus.cc:2804] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:25.982950 20472 raft_consensus.cc:697] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [term 1 LEADER]: Becoming Leader. State: Replica: 93f7c1b23fe64870a63b86600646ceff, State: Running, Role: LEADER
I20260501 02:02:25.983088 20472 consensus_queue.cc:237] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } }
I20260501 02:02:25.983750 20498 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: TxnStatusTablet state changed. Reason: New leader 93f7c1b23fe64870a63b86600646ceff. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:25.983832 20498 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:25.984054 20511 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260501 02:02:25.984305 20511 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260501 02:02:25.984514 19956 catalog_manager.cc:5671] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff reported cstate change: term changed from 0 to 1, leader changed from <none> to 93f7c1b23fe64870a63b86600646ceff (127.18.96.195). New cstate: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } health_report { overall_health: HEALTHY } } }
I20260501 02:02:25.997287 20505 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.994150 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:44955 (local address 127.0.0.1:51278)
0501 02:02:25.994278 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:25.994296 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:25.994385 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:25.994648 (+ 263us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:25.994651 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:25.994659 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:25.994957 (+ 298us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.994967 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.995716 (+ 749us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.995719 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:25.996184 (+ 465us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:25.996204 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.996307 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.996341 (+ 34us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:25.997037 (+ 696us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:25.997048 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:02:25.997114 (+ 66us) client_negotiation.cc:241] Negotiation successful
0501 02:02:25.997173 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260501 02:02:25.997330 20514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:25.994221 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.18.96.195:44955)
0501 02:02:25.994413 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:25.994416 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:25.994427 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:02:25.994477 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:25.994480 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:25.994525 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:25.994640 (+ 115us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:25.995088 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.995583 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:25.996317 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:25.996512 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:25.996564 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:25.996931 (+ 367us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:25.997030 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:25.997147 (+ 117us) server_negotiation.cc:300] Negotiation successful
0501 02:02:25.997191 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260501 02:02:26.000276 20100 raft_consensus.cc:1275] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Refusing update from remote peer 93f7c1b23fe64870a63b86600646ceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:26.000305 20239 raft_consensus.cc:1275] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f7c1b23fe64870a63b86600646ceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:26.000662 20498 consensus_queue.cc:1048] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:26.000790 20472 consensus_queue.cc:1048] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:26.002466 20476 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: TxnStatusTablet state changed. Reason: New leader 93f7c1b23fe64870a63b86600646ceff. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.002580 20476 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:26.002971 20478 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: TxnStatusTablet state changed. Reason: New leader 93f7c1b23fe64870a63b86600646ceff. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.003067 20478 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:26.003700 20498 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.003782 20498 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:26.004900 20476 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.004994 20476 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 7bee1dd6b9ff4a36be739fa3147fd517: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:26.005385 20472 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.005461 20472 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 93f7c1b23fe64870a63b86600646ceff: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:26.007454 20478 tablet_replica.cc:442] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "93f7c1b23fe64870a63b86600646ceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 } } peers { permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 } } peers { permanent_uuid: "93f7c1b23fe64870a63b86600646ceff" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 44955 } } }
I20260501 02:02:26.007546 20478 tablet_replica.cc:445] T 4ade33924972441e82a9c4b531512f55 P 5bd406cd16624132bc707efda1059ad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:26.009222 20517 mvcc.cc:204] Tried to move back new op lower bound from 7281053474811785216 to 7281053474748076032. Current Snapshot: MvccSnapshot[applied={T|T < 7281053474811785216}]
I20260501 02:02:26.013545 20516 mvcc.cc:204] Tried to move back new op lower bound from 7281053474811785216 to 7281053474748076032. Current Snapshot: MvccSnapshot[applied={T|T < 7281053474811785216}]
I20260501 02:02:26.016077 20514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:26.012976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51286 (local address 127.18.96.195:44955)
0501 02:02:26.013100 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:26.013104 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:26.013117 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:02:26.013155 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:26.013160 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:26.013208 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:26.013285 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:26.013750 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.014541 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:26.015410 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.015610 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:26.015651 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:26.015723 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:26.015800 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:26.015922 (+ 122us) server_negotiation.cc:300] Negotiation successful
0501 02:02:26.015962 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:02:26.025274 20528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:26.022333 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:44955 (local address 127.18.96.195:51837)
0501 02:02:26.022620 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:26.022632 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:26.022737 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:26.023016 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:26.023018 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:26.023025 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:26.023231 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:26.023235 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.024153 (+ 918us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:26.024156 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:26.024564 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:26.024569 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.024650 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:26.024677 (+ 27us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:26.025067 (+ 390us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:26.025072 (+ 5us) client_negotiation.cc:770] Sending connection context
0501 02:02:26.025120 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:02:26.025162 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260501 02:02:26.025300 20514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:26.022385 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:51837 (local address 127.18.96.195:44955)
0501 02:02:26.022498 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:26.022502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:26.022695 (+ 193us) server_negotiation.cc:408] Connection header received
0501 02:02:26.022819 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:26.022823 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:26.022894 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:26.022998 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:26.023335 (+ 337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.024052 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:26.024662 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.024848 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:26.024888 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:26.024955 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:26.025044 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:26.025153 (+ 109us) server_negotiation.cc:300] Negotiation successful
0501 02:02:26.025190 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:02:26.031926 20239 raft_consensus.cc:1275] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Refusing update from remote peer 93f7c1b23fe64870a63b86600646ceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:26.032267 20100 raft_consensus.cc:1275] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Refusing update from remote peer 93f7c1b23fe64870a63b86600646ceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:26.032364 20498 consensus_queue.cc:1048] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bd406cd16624132bc707efda1059ad3" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:26.032653 20498 consensus_queue.cc:1048] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bee1dd6b9ff4a36be739fa3147fd517" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:26.068974 19955 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:26.069123 19955 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:26.071084 19955 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 57a1be23c52a4e39a26c065e8fe5bcec: Sending DeleteTablet for 3 replicas of tablet 348c6a9af78944baaacff8c6697dbb0c
I20260501 02:02:26.071614 20237 tablet_service.cc:1558] Processing DeleteTablet for tablet 348c6a9af78944baaacff8c6697dbb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:26 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:40514
I20260501 02:02:26.071619 20095 tablet_service.cc:1558] Processing DeleteTablet for tablet 348c6a9af78944baaacff8c6697dbb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:26 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:46722
I20260501 02:02:26.071655 20379 tablet_service.cc:1558] Processing DeleteTablet for tablet 348c6a9af78944baaacff8c6697dbb0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:26 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:51276
I20260501 02:02:26.071951 20533 tablet_replica.cc:333] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: stopping tablet replica
I20260501 02:02:26.072098 20533 raft_consensus.cc:2243] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:26.072127 20535 tablet_replica.cc:333] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: stopping tablet replica
I20260501 02:02:26.072229 20533 raft_consensus.cc:2272] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:26.072393 20535 raft_consensus.cc:2243] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:26.072712 20535 raft_consensus.cc:2272] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:26.072914 20533 ts_tablet_manager.cc:1916] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:26.073634 20535 ts_tablet_manager.cc:1916] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:26.074132 20534 tablet_replica.cc:333] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: stopping tablet replica
I20260501 02:02:26.074256 20534 raft_consensus.cc:2243] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:26.074362 20534 raft_consensus.cc:2272] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:26.074900 20534 ts_tablet_manager.cc:1916] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:26.075629 20533 ts_tablet_manager.cc:1929] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:26.075731 20533 log.cc:1199] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/348c6a9af78944baaacff8c6697dbb0c
I20260501 02:02:26.076014 20533 ts_tablet_manager.cc:1950] T 348c6a9af78944baaacff8c6697dbb0c P 7bee1dd6b9ff4a36be739fa3147fd517: Deleting consensus metadata
I20260501 02:02:26.076751 20535 ts_tablet_manager.cc:1929] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:26.076740 19943 catalog_manager.cc:5002] TS 7bee1dd6b9ff4a36be739fa3147fd517 (127.18.96.193:37769): tablet 348c6a9af78944baaacff8c6697dbb0c (table test-table [id=7302df9894c24e2b9498447a7d697b63]) successfully deleted
I20260501 02:02:26.076905 20535 log.cc:1199] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/348c6a9af78944baaacff8c6697dbb0c
I20260501 02:02:26.077178 20535 ts_tablet_manager.cc:1950] T 348c6a9af78944baaacff8c6697dbb0c P 93f7c1b23fe64870a63b86600646ceff: Deleting consensus metadata
I20260501 02:02:26.077579 20534 ts_tablet_manager.cc:1929] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:26.077647 20534 log.cc:1199] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/348c6a9af78944baaacff8c6697dbb0c
I20260501 02:02:26.077732 19940 catalog_manager.cc:5002] TS 93f7c1b23fe64870a63b86600646ceff (127.18.96.195:44955): tablet 348c6a9af78944baaacff8c6697dbb0c (table test-table [id=7302df9894c24e2b9498447a7d697b63]) successfully deleted
I20260501 02:02:26.077924 20534 ts_tablet_manager.cc:1950] T 348c6a9af78944baaacff8c6697dbb0c P 5bd406cd16624132bc707efda1059ad3: Deleting consensus metadata
I20260501 02:02:26.078512 19942 catalog_manager.cc:5002] TS 5bd406cd16624132bc707efda1059ad3 (127.18.96.194:36991): tablet 348c6a9af78944baaacff8c6697dbb0c (table test-table [id=7302df9894c24e2b9498447a7d697b63]) successfully deleted
May 01 02:02:26 dist-test-slave-2j90 krb5kdc[19908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:26.086941 20465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:26.078293 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.18.96.193:37769)
0501 02:02:26.078416 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:26.078420 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:26.078658 (+ 238us) server_negotiation.cc:408] Connection header received
0501 02:02:26.078849 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:26.078852 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:26.078898 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:26.078991 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:26.079792 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.080270 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:26.080943 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.081186 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:26.083677 (+ 2491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:26.083696 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:26.083706 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:26.083728 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:26.085413 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:26.085843 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:26.085848 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:26.085850 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:26.085892 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:26.086138 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:26.086141 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:26.086142 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:26.086286 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:26.086416 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:26.086650 (+ 234us) server_negotiation.cc:300] Negotiation successful
0501 02:02:26.086739 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:02:26.087486 20138 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46734
I20260501 02:02:26.096243 20506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:26.089043 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.18.96.254:40761)
0501 02:02:26.089172 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:26.089176 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:26.089383 (+ 207us) server_negotiation.cc:408] Connection header received
0501 02:02:26.089478 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:26.089480 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:26.089535 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:26.089599 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:26.090470 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.090984 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:26.091775 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:26.091914 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:26.093006 (+ 1092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:26.093022 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:26.093024 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:26.093043 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:26.094613 (+ 1570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:26.095061 (+ 448us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:26.095064 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:26.095066 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:26.095106 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:26.095386 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:26.095389 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:26.095390 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:26.095517 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:26.095624 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:26.095916 (+ 292us) server_negotiation.cc:300] Negotiation successful
0501 02:02:26.096020 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:02:26.096632 19955 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58958
I20260501 02:02:26.097985 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20030
I20260501 02:02:26.106066 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20172
I20260501 02:02:26.112563 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20314
I20260501 02:02:26.119014 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 19924
2026-05-01T02:02:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3861 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:26 dist-test-slave-2j90 krb5kdc[20542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:26 dist-test-slave-2j90 krb5kdc[20542](info): set up 2 sockets
May 01 02:02:26 dist-test-slave-2j90 krb5kdc[20542](info): commencing operation
krb5kdc: starting...
W20260501 02:02:28.148474 18819 mini_kdc.cc:121] Time spent starting KDC: real 2.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260501 02:02:28.469152 18819 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.321s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260501 02:02:28.732651 18819 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.263s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260501 02:02:28.964570 18819 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.232s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/test-user.keytab.
W20260501 02:02:30.186371 18819 mini_kdc.cc:290] Time spent creating keytab for test-user: real 1.222s user 0.000s sys 0.002s
May 01 02:02:30 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
W20260501 02:02:30.685352 18819 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.18.96.254: real 0.479s user 0.000s sys 0.002s
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:30.708338 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37321
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:37321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:30.806072 20558 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:30.806275 20558 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:30.806356 20558 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:30.809484 20558 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:30.809562 20558 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260501 02:02:30.809597 20558 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:30.809664 20558 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:30.809705 20558 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:30.809738 20558 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:30.813746 20558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:37321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:30.814641 20558 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:30.815414 20558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:30.820621 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:30.820664 20558 server_base.cc:1061] running on GCE node
W20260501 02:02:30.820693 20563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:30.820899 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:30.821345 20558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:30.822194 20558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:30.823343 20558 hybrid_clock.cc:648] HybridClock initialized: now 1777600950823325 us; error 29 us; skew 500 ppm
May 01 02:02:30 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600950, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:30.825846 20558 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:30.826817 20558 webserver.cc:492] Webserver started at http://127.18.96.254:42163/ using document root <none> and password file <none>
I20260501 02:02:30.827347 20558 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:30.827421 20558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:30.827623 20558 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:30.829154 20558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "afaa50d2fcce4442b156ce9ac1e76e39"
format_stamp: "Formatted at 2026-05-01 02:02:30 on dist-test-slave-2j90"
server_key: "154401952de10bc8063ea963b252847e"
server_key_iv: "68533c4a3cb314669a33b05ee0ae7d64"
server_key_version: "encryptionkey@0"
I20260501 02:02:30.829653 20558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "afaa50d2fcce4442b156ce9ac1e76e39"
format_stamp: "Formatted at 2026-05-01 02:02:30 on dist-test-slave-2j90"
server_key: "154401952de10bc8063ea963b252847e"
server_key_iv: "68533c4a3cb314669a33b05ee0ae7d64"
server_key_version: "encryptionkey@0"
I20260501 02:02:30.832651 20558 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:30.834539 20573 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:30.835391 20558 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:30.835523 20558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "afaa50d2fcce4442b156ce9ac1e76e39"
format_stamp: "Formatted at 2026-05-01 02:02:30 on dist-test-slave-2j90"
server_key: "154401952de10bc8063ea963b252847e"
server_key_iv: "68533c4a3cb314669a33b05ee0ae7d64"
server_key_version: "encryptionkey@0"
I20260501 02:02:30.835618 20558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:30.863596 20558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:30.870311 20558 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:30.870509 20558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:30.878839 20558 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:37321
I20260501 02:02:30.878851 20635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:37321 every 8 connection(s)
I20260501 02:02:30.879803 20558 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:30.882817 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:30.883409 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20558
I20260501 02:02:30.883524 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:30.883822 18819 external_mini_cluster.cc:1468] Setting key 3f6e2bbf07cb21e22c1483499878ae54
I20260501 02:02:30.887914 20636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Bootstrap starting.
I20260501 02:02:30.889932 20636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:30.890751 20636 log.cc:826] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Log is configured to *not* fsync() on all Append() calls
May 01 02:02:30 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:30.892804 20636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: No bootstrap required, opened a new log
I20260501 02:02:30.895937 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:30.885024 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.18.96.254:37321)
0501 02:02:30.885457 (+ 433us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:30.885465 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:30.885494 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:02:30.886060 (+ 566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:30.886077 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:30.886325 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:30.886603 (+ 278us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:30.887446 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:30.888210 (+ 764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:30.888862 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:30.889141 (+ 279us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:30.891479 (+ 2338us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:30.891494 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:30.891504 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:30.891554 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:30.893434 (+ 1880us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:30.893826 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:30.893831 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:30.893836 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:30.893914 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:30.894170 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:30.894174 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:30.894176 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:30.894538 (+ 362us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:30.894706 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:30.894990 (+ 284us) server_negotiation.cc:300] Negotiation successful
0501 02:02:30.895252 (+ 262us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":136,"threads_started":1}
I20260501 02:02:30.896384 20636 raft_consensus.cc:359] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } }
I20260501 02:02:30.896564 20636 raft_consensus.cc:385] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:30.896627 20636 raft_consensus.cc:740] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afaa50d2fcce4442b156ce9ac1e76e39, State: Initialized, Role: FOLLOWER
I20260501 02:02:30.897089 20636 consensus_queue.cc:260] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } }
I20260501 02:02:30.897251 20636 raft_consensus.cc:399] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:30.897328 20636 raft_consensus.cc:493] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:30.897433 20636 raft_consensus.cc:3060] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:30.898391 20636 raft_consensus.cc:515] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } }
I20260501 02:02:30.898713 20636 leader_election.cc:304] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afaa50d2fcce4442b156ce9ac1e76e39; no voters:
I20260501 02:02:30.898999 20636 leader_election.cc:290] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:30.899076 20641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:30.899326 20641 raft_consensus.cc:697] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [term 1 LEADER]: Becoming Leader. State: Replica: afaa50d2fcce4442b156ce9ac1e76e39, State: Running, Role: LEADER
I20260501 02:02:30.899626 20641 consensus_queue.cc:237] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } }
I20260501 02:02:30.900066 20636 sys_catalog.cc:565] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:30.901321 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afaa50d2fcce4442b156ce9ac1e76e39. Latest consensus state: current_term: 1 leader_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } } }
I20260501 02:02:30.901320 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afaa50d2fcce4442b156ce9ac1e76e39" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37321 } } }
I20260501 02:02:30.901465 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:30.901465 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:30.901854 20650 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:30.904412 20650 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:30.909346 20650 catalog_manager.cc:1357] Generated new cluster ID: 925c21bb62c44c35a31eacd8db84aa97
I20260501 02:02:30.909413 20650 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:30.917675 20650 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:30.918524 20650 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:30.927233 20650 catalog_manager.cc:6044] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Generated new TSK 0
I20260501 02:02:30.927816 20650 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:30.981418 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37321
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:31.080916 20664 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:31.081075 20664 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:31.081159 20664 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:31.084395 20664 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:31.084446 20664 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:31.084493 20664 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260501 02:02:31.084525 20664 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:31.084564 20664 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:31.088506 20664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:31.089349 20664 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:31.090088 20664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:31.096240 20670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:31.096181 20669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:31.096184 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:31.096403 20664 server_base.cc:1061] running on GCE node
I20260501 02:02:31.096841 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:31.097414 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:31.098577 20664 hybrid_clock.cc:648] HybridClock initialized: now 1777600951098556 us; error 33 us; skew 500 ppm
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:31.101533 20664 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:31.102607 20664 webserver.cc:492] Webserver started at http://127.18.96.193:40311/ using document root <none> and password file <none>
I20260501 02:02:31.103155 20664 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:31.103231 20664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:31.103407 20664 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:31.105020 20664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "fb183c87de4facf6f8b9403b34dddbff"
server_key_iv: "3aa79a8b677722737393a1ba3183b00b"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.105487 20664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "fb183c87de4facf6f8b9403b34dddbff"
server_key_iv: "3aa79a8b677722737393a1ba3183b00b"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.108863 20664 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260501 02:02:31.110971 20679 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.111853 20664 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:31.112003 20664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "fb183c87de4facf6f8b9403b34dddbff"
server_key_iv: "3aa79a8b677722737393a1ba3183b00b"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.112099 20664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:31.123093 20664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:31.125229 20664 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:31.125408 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:31.126834 20664 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:31.126894 20664 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.126943 20664 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:31.126979 20664 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:31.139096 20664 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:32801
I20260501 02:02:31.139212 20794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:32801 every 8 connection(s)
I20260501 02:02:31.139979 20664 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:31.140545 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.127449 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39043 (local address 127.18.96.254:37321)
0501 02:02:31.127643 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.127651 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.128408 (+ 757us) server_negotiation.cc:408] Connection header received
0501 02:02:31.129589 (+ 1181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.129594 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.129712 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.129802 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.131444 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.132239 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.133287 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.133550 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.136680 (+ 3130us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.136705 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.136709 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.136743 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.138545 (+ 1802us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.139155 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.139160 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.139161 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.139216 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.139565 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.139567 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.139569 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.139698 (+ 129us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.139817 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.140265 (+ 448us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.140380 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260501 02:02:31.141045 20688 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.127564 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.193:39043)
0501 02:02:31.128215 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.128259 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.129349 (+ 1090us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.130102 (+ 753us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.130115 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.130636 (+ 521us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.131275 (+ 639us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.131292 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.132400 (+ 1108us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.132406 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.133108 (+ 702us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.133120 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.133764 (+ 644us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.134357 (+ 593us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.134386 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.136517 (+ 2131us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.138692 (+ 2175us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.138697 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.138707 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.139048 (+ 341us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.139321 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.139324 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.139327 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.139434 (+ 107us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.139818 (+ 384us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.139824 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.140017 (+ 193us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.140217 (+ 200us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.140422 (+ 205us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":113,"threads_started":1}
I20260501 02:02:31.142110 20795 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37321
I20260501 02:02:31.142324 20795 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:31.142753 20795 heartbeater.cc:507] Master 127.18.96.254:37321 requested a full tablet report, sending...
I20260501 02:02:31.144289 20589 ts_manager.cc:194] Registered new tserver with Master: f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801)
I20260501 02:02:31.145753 20589 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:39043
I20260501 02:02:31.146973 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20664
I20260501 02:02:31.147112 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:31.147344 18819 external_mini_cluster.cc:1468] Setting key d13216adf46586dcd2936a111ef7f1d5
I20260501 02:02:31.151381 20801 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.144211 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.193:33227)
0501 02:02:31.144572 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.144586 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.144677 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.144905 (+ 228us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.144908 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.145099 (+ 191us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.145371 (+ 272us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.145378 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.146068 (+ 690us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.146072 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.146625 (+ 553us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.146632 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.146756 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.148129 (+ 1373us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.148140 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.148420 (+ 280us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.150239 (+ 1819us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.150245 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.150249 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.150438 (+ 189us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.150689 (+ 251us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.150691 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.150692 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.150732 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.151055 (+ 323us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.151057 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.151117 (+ 60us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.151175 (+ 58us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.151259 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":94,"threads_started":1}
I20260501 02:02:31.151677 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.144335 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:33227 (local address 127.18.96.254:37321)
0501 02:02:31.144465 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.144468 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.144658 (+ 190us) server_negotiation.cc:408] Connection header received
0501 02:02:31.144766 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.144769 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.144805 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.144867 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.145476 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.145916 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.146740 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.146926 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.148694 (+ 1768us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.148706 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.148708 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.148728 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.150099 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.150533 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.150538 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.150542 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.150594 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.150813 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.150816 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.150819 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.150967 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.151278 (+ 311us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.151350 (+ 72us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.151496 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:31.192649 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37321
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:31.290278 20806 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:31.290446 20806 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:31.290503 20806 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:31.293560 20806 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:31.293610 20806 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:31.293658 20806 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260501 02:02:31.293673 20806 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:31.293711 20806 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:31.297706 20806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:31.298658 20806 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:31.299413 20806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:31.305291 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:31.305342 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:31.305269 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:31.305506 20806 server_base.cc:1061] running on GCE node
I20260501 02:02:31.305891 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:31.306437 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:31.307585 20806 hybrid_clock.cc:648] HybridClock initialized: now 1777600951307557 us; error 40 us; skew 500 ppm
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:31.310045 20806 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:31.310945 20806 webserver.cc:492] Webserver started at http://127.18.96.194:38901/ using document root <none> and password file <none>
I20260501 02:02:31.311448 20806 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:31.311519 20806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:31.311717 20806 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:31.313308 20806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "d794bce3a2944d5697194d1d79613baf"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "8f7cbfeb56cdeeaa5a5834d5650603ef"
server_key_iv: "bece895254da7ef6f743554195895354"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.313797 20806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "d794bce3a2944d5697194d1d79613baf"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "8f7cbfeb56cdeeaa5a5834d5650603ef"
server_key_iv: "bece895254da7ef6f743554195895354"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.317201 20806 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:02:31.319197 20821 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.320132 20806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:31.320256 20806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "d794bce3a2944d5697194d1d79613baf"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "8f7cbfeb56cdeeaa5a5834d5650603ef"
server_key_iv: "bece895254da7ef6f743554195895354"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.320361 20806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:31.354784 20806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:31.357041 20806 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:31.357228 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:31.358719 20806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:31.358791 20806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.358860 20806 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:31.358914 20806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:31.369881 20806 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:35409
I20260501 02:02:31.369906 20936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:35409 every 8 connection(s)
I20260501 02:02:31.370514 20830 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.359522 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.194:54277)
0501 02:02:31.359921 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.359955 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.360713 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.361216 (+ 503us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.361224 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.361716 (+ 492us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.362329 (+ 613us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.362340 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.363110 (+ 770us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.363114 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.363522 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.363528 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.363717 (+ 189us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.364295 (+ 578us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.364321 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.366248 (+ 1927us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.368286 (+ 2038us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.368294 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.368305 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.368514 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.368805 (+ 291us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.368812 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.368813 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.368890 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.369252 (+ 362us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.369258 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.369450 (+ 192us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.369647 (+ 197us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.369896 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":119,"threads_started":1}
I20260501 02:02:31.370764 20806 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:31.371346 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.359213 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:54277 (local address 127.18.96.254:37321)
0501 02:02:31.359370 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.359375 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.360074 (+ 699us) server_negotiation.cc:408] Connection header received
0501 02:02:31.360900 (+ 826us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.360904 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.360971 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.361072 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.362475 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.362964 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.363638 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.363818 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.366399 (+ 2581us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.366417 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.366420 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.366448 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.368171 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.368642 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.368646 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.368648 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.368698 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.368987 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.368991 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.368993 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.369145 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.369254 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.371073 (+ 1819us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.371206 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:02:31.372294 20937 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37321
I20260501 02:02:31.372454 20937 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:31.372870 20937 heartbeater.cc:507] Master 127.18.96.254:37321 requested a full tablet report, sending...
I20260501 02:02:31.373833 20590 ts_manager.cc:194] Registered new tserver with Master: d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409)
I20260501 02:02:31.374565 20590 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:54277
I20260501 02:02:31.377836 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20806
I20260501 02:02:31.377965 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:31.378228 18819 external_mini_cluster.cc:1468] Setting key a55695c17ce7c48070721eff4f2c29c5
I20260501 02:02:31.381798 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.374372 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:54791 (local address 127.18.96.254:37321)
0501 02:02:31.374497 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.374500 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.374603 (+ 103us) server_negotiation.cc:408] Connection header received
0501 02:02:31.374784 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.374787 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.374849 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.374932 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.375584 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.376033 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.376645 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.376769 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.377444 (+ 675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.377457 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.377459 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.377479 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.378981 (+ 1502us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.379367 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.379372 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.379374 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.379420 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.381099 (+ 1679us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.381102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.381104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.381248 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.381362 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.381580 (+ 218us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.381673 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:02:31.381814 20943 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.374245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.194:54791)
0501 02:02:31.374534 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.374548 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.374648 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.374971 (+ 323us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.374973 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.375183 (+ 210us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.375467 (+ 284us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.375475 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.376144 (+ 669us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.376148 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.376550 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.376557 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.376644 (+ 87us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.377096 (+ 452us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.377107 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.377341 (+ 234us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.379099 (+ 1758us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.379104 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.379106 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.379269 (+ 163us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.380935 (+ 1666us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.380937 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.380939 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.380982 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.381351 (+ 369us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.381354 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.381429 (+ 75us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.381531 (+ 102us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.381645 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:31.423105 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37321
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:31.519600 20948 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:31.519788 20948 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:31.519845 20948 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:31.523018 20948 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:31.523069 20948 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:31.523169 20948 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260501 02:02:31.523203 20948 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:02:31.523274 20948 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:31.527298 20948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:31.528195 20948 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:31.528967 20948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:31.535091 20953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:31.535133 20954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:31.535235 20948 server_base.cc:1061] running on GCE node
W20260501 02:02:31.535092 20956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:31.535732 20948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:31.536242 20948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:31.537395 20948 hybrid_clock.cc:648] HybridClock initialized: now 1777600951537372 us; error 34 us; skew 500 ppm
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:31.539989 20948 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:31.540890 20948 webserver.cc:492] Webserver started at http://127.18.96.195:36751/ using document root <none> and password file <none>
I20260501 02:02:31.541360 20948 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:31.541404 20948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:31.541545 20948 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:31.543193 20948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "38670c6b486c42e9866b3ca9289a48c1"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "9aedef48c3c94aedd422781f6be7f5c9"
server_key_iv: "311595c1ac47ec6b1bca3ae878b13e51"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.543603 20948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "38670c6b486c42e9866b3ca9289a48c1"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "9aedef48c3c94aedd422781f6be7f5c9"
server_key_iv: "311595c1ac47ec6b1bca3ae878b13e51"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.546522 20948 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:02:31.548403 20963 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.549304 20948 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:31.549414 20948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "38670c6b486c42e9866b3ca9289a48c1"
format_stamp: "Formatted at 2026-05-01 02:02:31 on dist-test-slave-2j90"
server_key: "9aedef48c3c94aedd422781f6be7f5c9"
server_key_iv: "311595c1ac47ec6b1bca3ae878b13e51"
server_key_version: "encryptionkey@0"
I20260501 02:02:31.549502 20948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:31.564486 20948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:31.566848 20948 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:31.567055 20948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:31.568462 20948 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:31.568509 20948 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:31.568545 20948 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:31.568609 20948 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:31.579874 20948 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:43197
I20260501 02:02:31.579903 21078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:43197 every 8 connection(s)
I20260501 02:02:31.580498 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.569156 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:41051 (local address 127.18.96.254:37321)
0501 02:02:31.569306 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.569311 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.569937 (+ 626us) server_negotiation.cc:408] Connection header received
0501 02:02:31.570786 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.570790 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.570850 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.570956 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.572131 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.572900 (+ 769us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.573853 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.574124 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.576778 (+ 2654us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.576803 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.576808 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.576844 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.578445 (+ 1601us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.579023 (+ 578us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.579028 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.579030 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.579084 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.579382 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.579386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.579388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.579569 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.579670 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.580233 (+ 563us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.580348 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:02:31.580924 20948 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:31.580951 20972 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.569400 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.195:41051)
0501 02:02:31.569779 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.569815 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.570599 (+ 784us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.571098 (+ 499us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.571105 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.571568 (+ 463us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.571982 (+ 414us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.571992 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.573012 (+ 1020us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.573017 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.573679 (+ 662us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.573687 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.573917 (+ 230us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.574546 (+ 629us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.574573 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.576633 (+ 2060us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.578555 (+ 1922us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.578560 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.578571 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.578898 (+ 327us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.579181 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.579184 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.579186 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.579288 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.579724 (+ 436us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.579729 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.579969 (+ 240us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.580161 (+ 192us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.580326 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
I20260501 02:02:31.582230 21079 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37321
I20260501 02:02:31.582404 21079 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:31.582926 21079 heartbeater.cc:507] Master 127.18.96.254:37321 requested a full tablet report, sending...
I20260501 02:02:31.583920 20590 ts_manager.cc:194] Registered new tserver with Master: 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:31.584673 20590 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:41051
I20260501 02:02:31.587543 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 20948
I20260501 02:02:31.587663 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:31.587935 18819 external_mini_cluster.cc:1468] Setting key b0c7c562e9e360c7fe08523541cddfe3
I20260501 02:02:31.589985 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:02:31.590842 21085 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.584244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.18.96.195:51115)
0501 02:02:31.584509 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.584522 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.584599 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.585213 (+ 614us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.585217 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.585406 (+ 189us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.585656 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.585664 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.586378 (+ 714us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.586385 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.586744 (+ 359us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.586750 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.586833 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.587215 (+ 382us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.587226 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.587597 (+ 371us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.589569 (+ 1972us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.589573 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.589575 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.589813 (+ 238us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.590107 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.590111 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.590113 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.590170 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.590505 (+ 335us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.590507 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.590567 (+ 60us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.590643 (+ 76us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.590725 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260501 02:02:31.591037 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.584931 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:51115 (local address 127.18.96.254:37321)
0501 02:02:31.585057 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.585060 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.585071 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:02:31.585097 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.585099 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.585130 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.585191 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.585770 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.586272 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.586831 (+ 559us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.586937 (+ 106us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.587880 (+ 943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.587901 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.587903 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.587924 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.589465 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.589926 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.589929 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.589931 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.590007 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.590257 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.590259 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.590261 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.590409 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.590513 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.590724 (+ 211us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.590829 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:31.612092 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.605603 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.18.96.254:37321)
0501 02:02:31.605724 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.605727 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.605737 (+ 10us) server_negotiation.cc:408] Connection header received
0501 02:02:31.605766 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.605768 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.605800 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.605862 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.606618 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.607089 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.607669 (+ 580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.607786 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.609717 (+ 1931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.609731 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.609733 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.609765 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.610964 (+ 1199us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.611325 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.611328 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.611329 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.611364 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.611544 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.611545 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.611547 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.611654 (+ 107us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.611748 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.611913 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.611985 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:02:31.614454 20590 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:31.616746 20590 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:31.628907 21098 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.624243 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.0.0.1:38948)
0501 02:02:31.624718 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.624754 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.624873 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.625782 (+ 909us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.625786 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.625803 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:31.626082 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.626091 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.627479 (+ 1388us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.627483 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.628405 (+ 922us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.628414 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.628553 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.628611 (+ 58us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.628700 (+ 89us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.628769 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":82,"threads_started":1}
I20260501 02:02:31.629211 21096 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.624412 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43197 (local address 127.0.0.1:60970)
0501 02:02:31.624718 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.624755 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.624872 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.626514 (+ 1642us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.626517 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.626540 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:31.626777 (+ 237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.626784 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.628085 (+ 1301us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.628089 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.628919 (+ 830us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.628926 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.628992 (+ 66us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.629003 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.629036 (+ 33us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.629078 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260501 02:02:31.629536 21097 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.624357 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.18.96.193:32801)
0501 02:02:31.625324 (+ 967us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.625330 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.625351 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:02:31.625439 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.625447 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.625638 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.625788 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:31.626221 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.627357 (+ 1136us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.628561 (+ 1204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.629115 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.629227 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.629298 (+ 71us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.629376 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":55,"threads_started":1}
I20260501 02:02:31.629083 21099 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.624891 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35409 (local address 127.0.0.1:50372)
0501 02:02:31.625525 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.625537 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.625608 (+ 71us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.626368 (+ 760us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.626373 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.626392 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:31.626667 (+ 275us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.626675 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.627818 (+ 1143us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.627822 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.628718 (+ 896us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.628727 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.628827 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.628840 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.628885 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.628934 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":100,"threads_started":1}
I20260501 02:02:31.630119 21101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.625338 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.18.96.195:43197)
0501 02:02:31.626120 (+ 782us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.626128 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.626148 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:02:31.626219 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.626224 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.626363 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.626513 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:31.626911 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.627969 (+ 1058us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.629202 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.629722 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.629819 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.629880 (+ 61us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.629986 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":104,"threads_started":1}
I20260501 02:02:31.630206 21100 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.625241 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.18.96.194:35409)
0501 02:02:31.625792 (+ 551us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.625797 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.625815 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:02:31.625895 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.625899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.626072 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.626202 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:31.626813 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.627600 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.629365 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.629822 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.629900 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.629989 (+ 89us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.630073 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":116,"threads_started":1}
I20260501 02:02:31.632315 21013 tablet_service.cc:1511] Processing CreateTablet for tablet 73e497a0b8784f6f92f06a190c90a782 (DEFAULT_TABLE table=test-table [id=32593c6a2acf4234ad6e918b75466b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:31.632368 20719 tablet_service.cc:1511] Processing CreateTablet for tablet 73e497a0b8784f6f92f06a190c90a782 (DEFAULT_TABLE table=test-table [id=32593c6a2acf4234ad6e918b75466b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:31.633175 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e497a0b8784f6f92f06a190c90a782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.632964 20871 tablet_service.cc:1511] Processing CreateTablet for tablet 73e497a0b8784f6f92f06a190c90a782 (DEFAULT_TABLE table=test-table [id=32593c6a2acf4234ad6e918b75466b37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:31.633276 20719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e497a0b8784f6f92f06a190c90a782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.633760 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73e497a0b8784f6f92f06a190c90a782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.637813 21102 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Bootstrap starting.
I20260501 02:02:31.639356 21103 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Bootstrap starting.
I20260501 02:02:31.639737 21104 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Bootstrap starting.
I20260501 02:02:31.640134 21102 tablet_bootstrap.cc:654] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.641155 21102 log.cc:826] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:31.641420 21104 tablet_bootstrap.cc:654] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.642539 21104 log.cc:826] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:31.642935 21102 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: No bootstrap required, opened a new log
I20260501 02:02:31.643113 21102 ts_tablet_manager.cc:1403] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:02:31.643831 21103 tablet_bootstrap.cc:654] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.644361 21103 log.cc:826] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:31.645748 21103 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: No bootstrap required, opened a new log
I20260501 02:02:31.645778 21102 raft_consensus.cc:359] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.645920 21103 ts_tablet_manager.cc:1403] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260501 02:02:31.646008 21102 raft_consensus.cc:385] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.646075 21102 raft_consensus.cc:740] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.646543 21102 consensus_queue.cc:260] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.647241 21079 heartbeater.cc:499] Master 127.18.96.254:37321 was elected leader, sending a full tablet report...
I20260501 02:02:31.647325 21104 tablet_bootstrap.cc:492] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: No bootstrap required, opened a new log
I20260501 02:02:31.647392 21102 ts_tablet_manager.cc:1434] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260501 02:02:31.647476 21104 ts_tablet_manager.cc:1403] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260501 02:02:31.649480 21103 raft_consensus.cc:359] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.649703 21103 raft_consensus.cc:385] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.649813 21103 raft_consensus.cc:740] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.650007 21104 raft_consensus.cc:359] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.650226 21104 raft_consensus.cc:385] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.650198 21103 consensus_queue.cc:260] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.650287 21104 raft_consensus.cc:740] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.650699 21104 consensus_queue.cc:260] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.650900 20795 heartbeater.cc:499] Master 127.18.96.254:37321 was elected leader, sending a full tablet report...
I20260501 02:02:31.651007 21103 ts_tablet_manager.cc:1434] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260501 02:02:31.651310 20937 heartbeater.cc:499] Master 127.18.96.254:37321 was elected leader, sending a full tablet report...
I20260501 02:02:31.651543 21104 ts_tablet_manager.cc:1434] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260501 02:02:31.824062 21110 raft_consensus.cc:493] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:31.824270 21110 raft_consensus.cc:515] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.825294 21110 leader_election.cc:290] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:31.828001 20830 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.825531 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43197 (local address 127.18.96.194:39939)
0501 02:02:31.825731 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.825745 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.825844 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.826134 (+ 290us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.826137 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.826157 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:31.826364 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.826369 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.827092 (+ 723us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.827094 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.827629 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.827635 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.827760 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.827776 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.827825 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.827877 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260501 02:02:31.828207 21111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.825531 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.18.96.194:40987)
0501 02:02:31.825892 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.825903 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.826053 (+ 150us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.826337 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.826341 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.826370 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:31.826602 (+ 232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.826607 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.827284 (+ 677us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.827288 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.827827 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.827833 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.827955 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.827969 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.828018 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.828073 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":63,"thread_start_us":94,"threads_started":1}
I20260501 02:02:31.828473 21101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.825642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:39939 (local address 127.18.96.195:43197)
0501 02:02:31.825758 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.825764 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.825782 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:02:31.825972 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.825976 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.826034 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.826120 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:31.826468 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.826965 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.827742 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.828209 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.828241 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.828287 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.828344 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:02:31.828720 21097 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.825633 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40987 (local address 127.18.96.193:32801)
0501 02:02:31.825763 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.825767 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.825979 (+ 212us) server_negotiation.cc:408] Connection header received
0501 02:02:31.826177 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.826179 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.826222 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.826327 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:31.826728 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.827166 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.827943 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.828427 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.828465 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.828521 (+ 56us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.828583 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:02:31.829131 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e497a0b8784f6f92f06a190c90a782" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1" is_pre_election: true
I20260501 02:02:31.829334 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e497a0b8784f6f92f06a190c90a782" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" is_pre_election: true
I20260501 02:02:31.829403 21033 raft_consensus.cc:2468] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:31.829571 20746 raft_consensus.cc:2468] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:31.829905 20822 leader_election.cc:304] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, d794bce3a2944d5697194d1d79613baf; no voters:
I20260501 02:02:31.830199 21110 raft_consensus.cc:2804] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:31.830265 21110 raft_consensus.cc:493] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:31.830307 21110 raft_consensus.cc:3060] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:31.831240 21110 raft_consensus.cc:515] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.831543 21110 leader_election.cc:290] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Requested vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:31.831856 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e497a0b8784f6f92f06a190c90a782" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
W20260501 02:02:31.831892 21080 tablet.cc:2404] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:31.831920 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73e497a0b8784f6f92f06a190c90a782" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1"
I20260501 02:02:31.831979 20746 raft_consensus.cc:3060] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:31.832053 21033 raft_consensus.cc:3060] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:31.832990 20746 raft_consensus.cc:2468] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:31.833132 21033 raft_consensus.cc:2468] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:31.833315 20825 leader_election.cc:304] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d794bce3a2944d5697194d1d79613baf, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:31.833495 21110 raft_consensus.cc:2804] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:31.833679 21110 raft_consensus.cc:697] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Becoming Leader. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Running, Role: LEADER
I20260501 02:02:31.834014 21110 consensus_queue.cc:237] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } }
I20260501 02:02:31.837073 20588 catalog_manager.cc:5671] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf reported cstate change: term changed from 0 to 1, leader changed from <none> to d794bce3a2944d5697194d1d79613baf (127.18.96.194). New cstate: current_term: 1 leader_uuid: "d794bce3a2944d5697194d1d79613baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } health_report { overall_health: HEALTHY } } }
W20260501 02:02:31.871814 20938 tablet.cc:2404] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
May 01 02:02:31 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600950, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
W20260501 02:02:31.891414 20796 tablet.cc:2404] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:31.891986 21119 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.883843 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37321 (local address 127.0.0.1:33540)
0501 02:02:31.884187 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:31.884199 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:31.884329 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:31.884631 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:31.884637 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:31.885022 (+ 385us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:31.885224 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.885229 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.886242 (+ 1013us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.886244 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:31.886609 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:31.886614 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.886705 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.887254 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:31.887267 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:31.888810 (+ 1543us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:31.890569 (+ 1759us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.890576 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.890580 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.890785 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.891010 (+ 225us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:31.891012 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:31.891013 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:31.891059 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:31.891410 (+ 351us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:31.891415 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:31.891523 (+ 108us) client_negotiation.cc:770] Sending connection context
0501 02:02:31.891668 (+ 145us) client_negotiation.cc:241] Negotiation successful
0501 02:02:31.891805 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260501 02:02:31.892023 20639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:31.883893 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.18.96.254:37321)
0501 02:02:31.884036 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:31.884040 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:31.884278 (+ 238us) server_negotiation.cc:408] Connection header received
0501 02:02:31.884431 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:31.884435 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:31.884494 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:31.884585 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:31.885346 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.886143 (+ 797us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:31.886713 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:31.886891 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:31.888925 (+ 2034us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:31.888942 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:31.888946 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:31.888970 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:31.890455 (+ 1485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.890877 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.890880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.890882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.890925 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:31.891135 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:31.891138 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:31.891140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:31.891289 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:31.891392 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:31.891741 (+ 349us) server_negotiation.cc:300] Negotiation successful
0501 02:02:31.891871 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":52}
I20260501 02:02:31.895713 20588 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:33540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260501 02:02:31.896415 20588 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:31.901260 20719 tablet_service.cc:1511] Processing CreateTablet for tablet f42db7b9047f4f0e87a2d5a8193a1305 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9b4769397aa48bbbfeddddb332bbf7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:31.901417 20871 tablet_service.cc:1511] Processing CreateTablet for tablet f42db7b9047f4f0e87a2d5a8193a1305 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9b4769397aa48bbbfeddddb332bbf7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:31.901611 21013 tablet_service.cc:1511] Processing CreateTablet for tablet f42db7b9047f4f0e87a2d5a8193a1305 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9b4769397aa48bbbfeddddb332bbf7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:02:31.902024 20719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f42db7b9047f4f0e87a2d5a8193a1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.902096 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f42db7b9047f4f0e87a2d5a8193a1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.902096 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f42db7b9047f4f0e87a2d5a8193a1305. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:31.904422 21103 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: Bootstrap starting.
I20260501 02:02:31.904645 21104 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: Bootstrap starting.
I20260501 02:02:31.904753 21102 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: Bootstrap starting.
I20260501 02:02:31.905524 21103 tablet_bootstrap.cc:654] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.905632 21104 tablet_bootstrap.cc:654] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.905819 21102 tablet_bootstrap.cc:654] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:31.907065 21102 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: No bootstrap required, opened a new log
I20260501 02:02:31.907076 21103 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: No bootstrap required, opened a new log
I20260501 02:02:31.907156 21103 ts_tablet_manager.cc:1403] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:31.907156 21102 ts_tablet_manager.cc:1403] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:31.907660 21103 raft_consensus.cc:359] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.907775 21103 raft_consensus.cc:385] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.907825 21103 raft_consensus.cc:740] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.907785 21102 raft_consensus.cc:359] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.907900 21102 raft_consensus.cc:385] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.907931 21102 raft_consensus.cc:740] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.907963 21103 consensus_queue.cc:260] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.908051 21102 consensus_queue.cc:260] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.908239 21103 ts_tablet_manager.cc:1434] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:31.908288 21102 ts_tablet_manager.cc:1434] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:31.908470 21109 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:31.908486 21108 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:31.908589 21108 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:31.908588 21109 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:31.909457 21104 tablet_bootstrap.cc:492] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: No bootstrap required, opened a new log
I20260501 02:02:31.909566 21104 ts_tablet_manager.cc:1403] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260501 02:02:31.910056 21104 raft_consensus.cc:359] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.910163 21104 raft_consensus.cc:385] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:31.910202 21104 raft_consensus.cc:740] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Initialized, Role: FOLLOWER
I20260501 02:02:31.910354 21104 consensus_queue.cc:260] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:31.910754 21104 ts_tablet_manager.cc:1434] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260501 02:02:31.910712 21110 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:31.910827 21110 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:32.158963 21109 raft_consensus.cc:493] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:32.159107 21109 raft_consensus.cc:515] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.160140 21109 leader_election.cc:290] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:32.163161 21123 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.160349 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35409 (local address 127.18.96.193:33489)
0501 02:02:32.160597 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:32.160615 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:32.160773 (+ 158us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:32.161088 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:32.161091 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:32.161111 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:32.161333 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.161339 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.162160 (+ 821us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.162166 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:32.162752 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.162759 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.162891 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.162908 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:02:32.162959 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:02:32.163013 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260501 02:02:32.163635 21124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.160467 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:33489 (local address 127.18.96.194:35409)
0501 02:02:32.160753 (+ 286us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.160759 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.160780 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:02:32.160916 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.160920 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.160973 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.161083 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:32.161456 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.162041 (+ 585us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.162875 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.163370 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.163404 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.163453 (+ 49us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.163500 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":71,"threads_started":1}
I20260501 02:02:32.163791 21125 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.160715 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43197 (local address 127.18.96.193:41227)
0501 02:02:32.160935 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:32.160948 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:32.161055 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:32.161340 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:32.161343 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:32.161359 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:32.161996 (+ 637us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.162012 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.162701 (+ 689us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.162705 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:32.163368 (+ 663us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.163377 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.163544 (+ 167us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.163559 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:02:32.163610 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:02:32.163657 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":49,"threads_started":1}
I20260501 02:02:32.164173 20891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f42db7b9047f4f0e87a2d5a8193a1305" candidate_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d794bce3a2944d5697194d1d79613baf" is_pre_election: true
I20260501 02:02:32.164258 21101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.160836 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:41227 (local address 127.18.96.195:43197)
0501 02:02:32.160958 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.160964 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.161027 (+ 63us) server_negotiation.cc:408] Connection header received
0501 02:02:32.161174 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.161184 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.161226 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.161334 (+ 108us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:32.162125 (+ 791us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.162586 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.163531 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.164013 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.164044 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.164094 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.164140 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:02:32.164402 20891 raft_consensus.cc:2468] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd416d8d7041bcb35b0b0436fdf1f5 in term 0.
I20260501 02:02:32.164816 20680 leader_election.cc:304] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d794bce3a2944d5697194d1d79613baf, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:32.164824 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f42db7b9047f4f0e87a2d5a8193a1305" candidate_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1" is_pre_election: true
I20260501 02:02:32.164973 21033 raft_consensus.cc:2468] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5cd416d8d7041bcb35b0b0436fdf1f5 in term 0.
I20260501 02:02:32.165076 21109 raft_consensus.cc:2804] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:32.165167 21109 raft_consensus.cc:493] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:32.165223 21109 raft_consensus.cc:3060] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.166193 21109 raft_consensus.cc:515] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.166528 21109 leader_election.cc:290] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [CANDIDATE]: Term 1 election: Requested vote from peers d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:32.166919 20891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f42db7b9047f4f0e87a2d5a8193a1305" candidate_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d794bce3a2944d5697194d1d79613baf"
I20260501 02:02:32.166980 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f42db7b9047f4f0e87a2d5a8193a1305" candidate_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1"
I20260501 02:02:32.167060 20891 raft_consensus.cc:3060] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.167093 21033 raft_consensus.cc:3060] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.167902 21033 raft_consensus.cc:2468] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5cd416d8d7041bcb35b0b0436fdf1f5 in term 1.
I20260501 02:02:32.168032 20891 raft_consensus.cc:2468] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5cd416d8d7041bcb35b0b0436fdf1f5 in term 1.
I20260501 02:02:32.168190 20680 leader_election.cc:304] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:32.168481 21109 raft_consensus.cc:2804] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:32.168677 21109 raft_consensus.cc:697] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 LEADER]: Becoming Leader. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Running, Role: LEADER
I20260501 02:02:32.169014 21109 consensus_queue.cc:237] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.171372 21126 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: TxnStatusTablet state changed. Reason: New leader f5cd416d8d7041bcb35b0b0436fdf1f5. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.171476 21126 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:32.171758 21128 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260501 02:02:32.171970 21128 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260501 02:02:32.172622 20588 catalog_manager.cc:5671] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
I20260501 02:02:32.200302 21097 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.197277 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.18.96.193:32801)
0501 02:02:32.197398 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.197403 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.197417 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:02:32.197593 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.197596 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.197653 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.197749 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:32.198123 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.198602 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.199241 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.199439 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.199486 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.199834 (+ 348us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.199972 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.200114 (+ 142us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.200157 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:02:32.200302 21119 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.197164 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.0.0.1:38962)
0501 02:02:32.197291 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:32.197318 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:32.197433 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:32.197761 (+ 328us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:32.197764 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:32.197772 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:32.198001 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.198008 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.198714 (+ 706us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.198719 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:32.199126 (+ 407us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.199132 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.199254 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.199294 (+ 40us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.199942 (+ 648us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.199972 (+ 30us) client_negotiation.cc:770] Sending connection context
0501 02:02:32.200070 (+ 98us) client_negotiation.cc:241] Negotiation successful
0501 02:02:32.200152 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260501 02:02:32.205416 20891 raft_consensus.cc:1275] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Refusing update from remote peer f5cd416d8d7041bcb35b0b0436fdf1f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:32.205492 21033 raft_consensus.cc:1275] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Refusing update from remote peer f5cd416d8d7041bcb35b0b0436fdf1f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:32.206074 21126 consensus_queue.cc:1048] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:32.206306 21109 consensus_queue.cc:1048] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:32.209507 21110 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: TxnStatusTablet state changed. Reason: New leader f5cd416d8d7041bcb35b0b0436fdf1f5. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.209610 21110 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:32.210425 21108 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: TxnStatusTablet state changed. Reason: New leader f5cd416d8d7041bcb35b0b0436fdf1f5. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.210529 21108 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:32.212771 21109 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.212863 21109 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:32.214833 21126 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.214943 21126 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P f5cd416d8d7041bcb35b0b0436fdf1f5: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:02:32.217185 21110 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.217517 21110 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P d794bce3a2944d5697194d1d79613baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:32.217690 21108 tablet_replica.cc:442] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } } }
I20260501 02:02:32.217784 21108 tablet_replica.cc:445] T f42db7b9047f4f0e87a2d5a8193a1305 P 38670c6b486c42e9866b3ca9289a48c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:02:32.219996 21132 mvcc.cc:204] Tried to move back new op lower bound from 7281053500226969600 to 7281053500087169024. Current Snapshot: MvccSnapshot[applied={T|T < 7281053500226969600}]
I20260501 02:02:32.220191 21133 mvcc.cc:204] Tried to move back new op lower bound from 7281053500226969600 to 7281053500087169024. Current Snapshot: MvccSnapshot[applied={T|T < 7281053500226969600}]
I20260501 02:02:32.224884 21124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.220962 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.18.96.194:35409)
0501 02:02:32.221082 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.221086 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.221544 (+ 458us) server_negotiation.cc:408] Connection header received
0501 02:02:32.221697 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.221700 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.221748 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.221819 (+ 71us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:32.222327 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.223022 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.223829 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.224033 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.224089 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.224507 (+ 418us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.224609 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.224701 (+ 92us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.224760 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260501 02:02:32.235013 21097 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.231997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:45265 (local address 127.18.96.193:32801)
0501 02:02:32.232159 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.232162 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.232256 (+ 94us) server_negotiation.cc:408] Connection header received
0501 02:02:32.232398 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.232401 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.232446 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.232502 (+ 56us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:32.232915 (+ 413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.233377 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.234297 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.234513 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.234554 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.234619 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.234693 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.234837 (+ 144us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.234883 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260501 02:02:32.234967 21110 consensus_queue.cc:1048] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:32.235351 21144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.231863 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.18.96.194:45265)
0501 02:02:32.232170 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:32.232186 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:32.232286 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:32.232531 (+ 245us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:32.232534 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:32.232543 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:32.232808 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.232814 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.233505 (+ 691us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.233508 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:32.234170 (+ 662us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.234179 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.234294 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.234350 (+ 56us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.234722 (+ 372us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.234727 (+ 5us) client_negotiation.cc:770] Sending connection context
0501 02:02:32.235149 (+ 422us) client_negotiation.cc:241] Negotiation successful
0501 02:02:32.235212 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
I20260501 02:02:32.241138 21112 consensus_queue.cc:1048] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:32.263792 21124 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.260398 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:41585 (local address 127.18.96.194:35409)
0501 02:02:32.260652 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:32.260656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:32.260759 (+ 103us) server_negotiation.cc:408] Connection header received
0501 02:02:32.260897 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:32.260901 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:32.260950 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:32.261023 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:32.261415 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.262012 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.262904 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.263113 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.263152 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.263240 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.263319 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:32.263601 (+ 282us) server_negotiation.cc:300] Negotiation successful
0501 02:02:32.263646 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260501 02:02:32.263818 21149 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:32.260351 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35409 (local address 127.18.96.193:41585)
0501 02:02:32.260667 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:32.260684 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:32.260787 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:32.261048 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:32.261051 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:32.261059 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:32.261306 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.261312 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.262157 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:32.262162 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:32.262749 (+ 587us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:32.262758 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:32.262918 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:32.262957 (+ 39us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:32.263490 (+ 533us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:32.263498 (+ 8us) client_negotiation.cc:770] Sending connection context
0501 02:02:32.263591 (+ 93us) client_negotiation.cc:241] Negotiation successful
0501 02:02:32.263665 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":108,"threads_started":1}
I20260501 02:02:32.292336 20588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:32.292518 20588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:32.295336 20588 catalog_manager.cc:5958] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Sending DeleteTablet for 3 replicas of tablet 73e497a0b8784f6f92f06a190c90a782
I20260501 02:02:32.296039 20719 tablet_service.cc:1558] Processing DeleteTablet for tablet 73e497a0b8784f6f92f06a190c90a782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38948
I20260501 02:02:32.296061 21013 tablet_service.cc:1558] Processing DeleteTablet for tablet 73e497a0b8784f6f92f06a190c90a782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:60970
I20260501 02:02:32.296712 21152 tablet_replica.cc:333] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: stopping tablet replica
I20260501 02:02:32.296862 20871 tablet_service.cc:1558] Processing DeleteTablet for tablet 73e497a0b8784f6f92f06a190c90a782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:50372
I20260501 02:02:32.297000 21152 raft_consensus.cc:2243] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:32.297164 21152 raft_consensus.cc:2272] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:32.297489 21154 tablet_replica.cc:333] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: stopping tablet replica
I20260501 02:02:32.297753 21154 raft_consensus.cc:2243] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:32.298192 20588 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:32.298600 20588 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:32.299587 21154 raft_consensus.cc:2272] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:32.300853 21154 ts_tablet_manager.cc:1916] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:32.303738 21154 ts_tablet_manager.cc:1929] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:32.303874 21154 log.cc:1199] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/73e497a0b8784f6f92f06a190c90a782
I20260501 02:02:32.304111 21153 tablet_replica.cc:333] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: stopping tablet replica
I20260501 02:02:32.304314 21154 ts_tablet_manager.cc:1950] T 73e497a0b8784f6f92f06a190c90a782 P d794bce3a2944d5697194d1d79613baf: Deleting consensus metadata
I20260501 02:02:32.304421 21153 raft_consensus.cc:2243] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:32.304607 21153 raft_consensus.cc:2272] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:32.304759 20719 tablet_service.cc:1511] Processing CreateTablet for tablet 6b46dba7c832446ba16fff81569f2684 (DEFAULT_TABLE table=test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:32.305008 20719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b46dba7c832446ba16fff81569f2684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:32.305092 21013 tablet_service.cc:1511] Processing CreateTablet for tablet 6b46dba7c832446ba16fff81569f2684 (DEFAULT_TABLE table=test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:32.305316 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b46dba7c832446ba16fff81569f2684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:32.307360 21102 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Bootstrap starting.
I20260501 02:02:32.307468 21103 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Bootstrap starting.
I20260501 02:02:32.308089 21153 ts_tablet_manager.cc:1916] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:32.308205 21102 tablet_bootstrap.cc:654] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:32.308308 21103 tablet_bootstrap.cc:654] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:32.304802 20871 tablet_service.cc:1511] Processing CreateTablet for tablet 6b46dba7c832446ba16fff81569f2684 (DEFAULT_TABLE table=test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:32.309247 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b46dba7c832446ba16fff81569f2684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:32.310225 21152 ts_tablet_manager.cc:1916] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:32.310495 21153 ts_tablet_manager.cc:1929] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:32.310559 21153 log.cc:1199] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/73e497a0b8784f6f92f06a190c90a782
I20260501 02:02:32.310829 21153 ts_tablet_manager.cc:1950] T 73e497a0b8784f6f92f06a190c90a782 P 38670c6b486c42e9866b3ca9289a48c1: Deleting consensus metadata
I20260501 02:02:32.311429 21104 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Bootstrap starting.
I20260501 02:02:32.311940 20574 catalog_manager.cc:5002] TS 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197): tablet 73e497a0b8784f6f92f06a190c90a782 (table test-table [id=32593c6a2acf4234ad6e918b75466b37]) successfully deleted
I20260501 02:02:32.312453 21104 tablet_bootstrap.cc:654] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:32.314186 20574 catalog_manager.cc:5002] TS d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409): tablet 73e497a0b8784f6f92f06a190c90a782 (table test-table [id=32593c6a2acf4234ad6e918b75466b37]) successfully deleted
I20260501 02:02:32.314294 21103 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: No bootstrap required, opened a new log
I20260501 02:02:32.314383 21103 ts_tablet_manager.cc:1403] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260501 02:02:32.314790 21103 raft_consensus.cc:359] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.314880 21103 raft_consensus.cc:385] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:32.314906 21103 raft_consensus.cc:740] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Initialized, Role: FOLLOWER
I20260501 02:02:32.314922 21104 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: No bootstrap required, opened a new log
I20260501 02:02:32.314985 21104 ts_tablet_manager.cc:1403] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260501 02:02:32.315011 21103 consensus_queue.cc:260] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.315228 21103 ts_tablet_manager.cc:1434] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:32.315411 21104 raft_consensus.cc:359] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.315495 21104 raft_consensus.cc:385] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:32.315521 21104 raft_consensus.cc:740] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Initialized, Role: FOLLOWER
I20260501 02:02:32.315618 21104 consensus_queue.cc:260] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.315847 21104 ts_tablet_manager.cc:1434] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:32.317651 21102 tablet_bootstrap.cc:492] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: No bootstrap required, opened a new log
I20260501 02:02:32.317730 21102 ts_tablet_manager.cc:1403] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260501 02:02:32.318161 21102 raft_consensus.cc:359] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.318267 21102 raft_consensus.cc:385] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:32.318308 21102 raft_consensus.cc:740] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Initialized, Role: FOLLOWER
I20260501 02:02:32.318414 21102 consensus_queue.cc:260] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.318647 21102 ts_tablet_manager.cc:1434] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:32.320734 21152 ts_tablet_manager.cc:1929] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:32.320874 21152 log.cc:1199] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/73e497a0b8784f6f92f06a190c90a782
I20260501 02:02:32.321393 21152 ts_tablet_manager.cc:1950] T 73e497a0b8784f6f92f06a190c90a782 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting consensus metadata
I20260501 02:02:32.322585 20577 catalog_manager.cc:5002] TS f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801): tablet 73e497a0b8784f6f92f06a190c90a782 (table test-table [id=32593c6a2acf4234ad6e918b75466b37]) successfully deleted
I20260501 02:02:32.436477 21147 raft_consensus.cc:493] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:32.436625 21147 raft_consensus.cc:515] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.437156 21147 leader_election.cc:290] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:32.437594 21032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b46dba7c832446ba16fff81569f2684" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1" is_pre_election: true
I20260501 02:02:32.437595 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b46dba7c832446ba16fff81569f2684" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" is_pre_election: true
I20260501 02:02:32.437757 20746 raft_consensus.cc:2468] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:32.437762 21032 raft_consensus.cc:2468] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:32.438089 20822 leader_election.cc:304] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, d794bce3a2944d5697194d1d79613baf, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:32.438304 21147 raft_consensus.cc:2804] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:32.438385 21147 raft_consensus.cc:493] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:32.438423 21147 raft_consensus.cc:3060] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.439774 21147 raft_consensus.cc:515] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.440165 21147 leader_election.cc:290] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Requested vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:32.440503 21032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b46dba7c832446ba16fff81569f2684" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1"
I20260501 02:02:32.440502 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b46dba7c832446ba16fff81569f2684" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
I20260501 02:02:32.440613 20746 raft_consensus.cc:3060] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.440614 21032 raft_consensus.cc:3060] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:32.441455 21032 raft_consensus.cc:2468] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:32.441694 20746 raft_consensus.cc:2468] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:32.441761 20822 leader_election.cc:304] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, d794bce3a2944d5697194d1d79613baf; no voters:
I20260501 02:02:32.441984 21147 raft_consensus.cc:2804] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:32.442154 21147 raft_consensus.cc:697] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Becoming Leader. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Running, Role: LEADER
I20260501 02:02:32.442298 21147 consensus_queue.cc:237] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:32.444096 20590 catalog_manager.cc:5671] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf reported cstate change: term changed from 0 to 1, leader changed from <none> to d794bce3a2944d5697194d1d79613baf (127.18.96.194). New cstate: current_term: 1 leader_uuid: "d794bce3a2944d5697194d1d79613baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
I20260501 02:02:32.475872 20746 raft_consensus.cc:1275] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Refusing update from remote peer d794bce3a2944d5697194d1d79613baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:32.476143 21033 raft_consensus.cc:1275] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Refusing update from remote peer d794bce3a2944d5697194d1d79613baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:32.476300 21147 consensus_queue.cc:1048] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:32.476569 21112 consensus_queue.cc:1048] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 02:02:32.489882 21133 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260501 02:02:34.037175 20588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:34.037303 20588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:34.039706 20588 catalog_manager.cc:5958] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Sending DeleteTablet for 3 replicas of tablet 6b46dba7c832446ba16fff81569f2684
I20260501 02:02:34.040194 20719 tablet_service.cc:1558] Processing DeleteTablet for tablet 6b46dba7c832446ba16fff81569f2684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38948
I20260501 02:02:34.040316 20871 tablet_service.cc:1558] Processing DeleteTablet for tablet 6b46dba7c832446ba16fff81569f2684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:50372
I20260501 02:02:34.040496 21013 tablet_service.cc:1558] Processing DeleteTablet for tablet 6b46dba7c832446ba16fff81569f2684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:60970
I20260501 02:02:34.040557 21183 tablet_replica.cc:333] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: stopping tablet replica
I20260501 02:02:34.040711 21183 raft_consensus.cc:2243] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:34.040818 21183 raft_consensus.cc:2272] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:34.040827 21185 tablet_replica.cc:333] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: stopping tablet replica
I20260501 02:02:34.040930 21185 raft_consensus.cc:2243] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:34.041019 21185 raft_consensus.cc:2272] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:34.041565 21185 ts_tablet_manager.cc:1916] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.042816 21184 tablet_replica.cc:333] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: stopping tablet replica
I20260501 02:02:34.043015 21184 raft_consensus.cc:2243] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:34.043226 21184 raft_consensus.cc:2272] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:34.043772 21183 ts_tablet_manager.cc:1916] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.044220 21185 ts_tablet_manager.cc:1929] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260501 02:02:34.044312 21185 log.cc:1199] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/6b46dba7c832446ba16fff81569f2684
I20260501 02:02:34.044624 21185 ts_tablet_manager.cc:1950] T 6b46dba7c832446ba16fff81569f2684 P 38670c6b486c42e9866b3ca9289a48c1: Deleting consensus metadata
I20260501 02:02:34.045244 20574 catalog_manager.cc:5002] TS 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197): tablet 6b46dba7c832446ba16fff81569f2684 (table test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]) successfully deleted
I20260501 02:02:34.046427 21184 ts_tablet_manager.cc:1916] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.046793 21183 ts_tablet_manager.cc:1929] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260501 02:02:34.046867 21183 log.cc:1199] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/6b46dba7c832446ba16fff81569f2684
I20260501 02:02:34.047102 21183 ts_tablet_manager.cc:1950] T 6b46dba7c832446ba16fff81569f2684 P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting consensus metadata
I20260501 02:02:34.047778 20577 catalog_manager.cc:5002] TS f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801): tablet 6b46dba7c832446ba16fff81569f2684 (table test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]) successfully deleted
I20260501 02:02:34.048659 21184 ts_tablet_manager.cc:1929] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260501 02:02:34.048718 21184 log.cc:1199] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/6b46dba7c832446ba16fff81569f2684
I20260501 02:02:34.048914 21184 ts_tablet_manager.cc:1950] T 6b46dba7c832446ba16fff81569f2684 P d794bce3a2944d5697194d1d79613baf: Deleting consensus metadata
I20260501 02:02:34.049434 20574 catalog_manager.cc:5002] TS d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409): tablet 6b46dba7c832446ba16fff81569f2684 (table test-table [id=460d5b8b3f8c42d1ae5b9c63eae3e862]) successfully deleted
May 01 02:02:34 dist-test-slave-2j90 krb5kdc[20542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600954, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:02:34 dist-test-slave-2j90 krb5kdc[20542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600954, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:34.066555 21193 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.058875 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.18.96.254:37321)
0501 02:02:34.059137 (+ 262us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.059142 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.059253 (+ 111us) server_negotiation.cc:408] Connection header received
0501 02:02:34.059415 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.059419 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.059471 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.059564 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:34.060329 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.060849 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.061480 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.061692 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.063724 (+ 2032us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:34.063745 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:34.063747 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:34.063768 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:34.065030 (+ 1262us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:34.065411 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:34.065417 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:34.065423 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:34.065463 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:34.065685 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:34.065689 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:34.065691 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:34.065839 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:34.066014 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.066228 (+ 214us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.066356 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260501 02:02:34.068679 20588 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:34.069109 20588 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:34.073714 20719 tablet_service.cc:1511] Processing CreateTablet for tablet bed442a93bdb45949ebea110e3faeb5e (DEFAULT_TABLE table=test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.073895 20871 tablet_service.cc:1511] Processing CreateTablet for tablet bed442a93bdb45949ebea110e3faeb5e (DEFAULT_TABLE table=test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.074041 20719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bed442a93bdb45949ebea110e3faeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.074153 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bed442a93bdb45949ebea110e3faeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.075126 21013 tablet_service.cc:1511] Processing CreateTablet for tablet bed442a93bdb45949ebea110e3faeb5e (DEFAULT_TABLE table=test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.075397 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bed442a93bdb45949ebea110e3faeb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.076613 21194 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Bootstrap starting.
I20260501 02:02:34.077390 21194 tablet_bootstrap.cc:654] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.077458 21196 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Bootstrap starting.
I20260501 02:02:34.078115 21195 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Bootstrap starting.
I20260501 02:02:34.078650 21196 tablet_bootstrap.cc:654] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.078820 21194 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: No bootstrap required, opened a new log
I20260501 02:02:34.079067 21195 tablet_bootstrap.cc:654] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.079214 21194 ts_tablet_manager.cc:1403] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:34.079715 21194 raft_consensus.cc:359] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.079881 21194 raft_consensus.cc:385] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.079934 21194 raft_consensus.cc:740] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.079969 21196 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: No bootstrap required, opened a new log
I20260501 02:02:34.080065 21196 ts_tablet_manager.cc:1403] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260501 02:02:34.080124 21194 consensus_queue.cc:260] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.080309 21195 tablet_bootstrap.cc:492] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: No bootstrap required, opened a new log
I20260501 02:02:34.080379 21195 ts_tablet_manager.cc:1403] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:34.080395 21194 ts_tablet_manager.cc:1434] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:34.080504 21196 raft_consensus.cc:359] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.080586 21196 raft_consensus.cc:385] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.080607 21196 raft_consensus.cc:740] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.080719 21196 consensus_queue.cc:260] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.080847 21195 raft_consensus.cc:359] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.080981 21195 raft_consensus.cc:385] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.081017 21195 raft_consensus.cc:740] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.081055 21196 ts_tablet_manager.cc:1434] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260501 02:02:34.081171 21195 consensus_queue.cc:260] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.082603 21195 ts_tablet_manager.cc:1434] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:34.365561 21200 raft_consensus.cc:493] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:34.365688 21200 raft_consensus.cc:515] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.366875 21200 leader_election.cc:290] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409)
I20260501 02:02:34.370771 21202 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.367088 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35409 (local address 127.18.96.195:35161)
0501 02:02:34.367535 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:34.367547 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:34.367675 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:34.368011 (+ 336us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:34.368014 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:34.368051 (+ 37us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:34.368374 (+ 323us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.368381 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.369429 (+ 1048us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.369433 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:34.370353 (+ 920us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.370361 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.370472 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.370488 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:02:34.370536 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:02:34.370591 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":61,"spinlock_wait_cycles":34816,"thread_start_us":63,"threads_started":1}
I20260501 02:02:34.371179 21203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.367175 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:55121 (local address 127.18.96.193:32801)
0501 02:02:34.367509 (+ 334us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.367514 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.367632 (+ 118us) server_negotiation.cc:408] Connection header received
0501 02:02:34.367825 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.367828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.367884 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.367982 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:34.368504 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.369322 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.370492 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.370934 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.370964 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.371010 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.371052 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260501 02:02:34.371320 21201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.367091 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.18.96.195:55121)
0501 02:02:34.367519 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:34.367535 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:34.367674 (+ 139us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:34.368011 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:34.368014 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:34.368050 (+ 36us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:34.368374 (+ 324us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.368381 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.369429 (+ 1048us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.369433 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:34.370355 (+ 922us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.370361 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.370471 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.370488 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:02:34.370536 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:02:34.370591 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"spinlock_wait_cycles":23040,"thread_start_us":128,"threads_started":1}
I20260501 02:02:34.371344 21204 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.367187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:35161 (local address 127.18.96.194:35409)
0501 02:02:34.367509 (+ 322us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.367514 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.367632 (+ 118us) server_negotiation.cc:408] Connection header received
0501 02:02:34.367825 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.367828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.367884 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.367982 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:34.368504 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.369311 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.370664 (+ 1353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.371088 (+ 424us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.371124 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.371183 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.371240 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260501 02:02:34.371675 20891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bed442a93bdb45949ebea110e3faeb5e" candidate_uuid: "38670c6b486c42e9866b3ca9289a48c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d794bce3a2944d5697194d1d79613baf" is_pre_election: true
I20260501 02:02:34.371840 20891 raft_consensus.cc:2468] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38670c6b486c42e9866b3ca9289a48c1 in term 0.
I20260501 02:02:34.371824 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bed442a93bdb45949ebea110e3faeb5e" candidate_uuid: "38670c6b486c42e9866b3ca9289a48c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" is_pre_election: true
I20260501 02:02:34.371939 20746 raft_consensus.cc:2468] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38670c6b486c42e9866b3ca9289a48c1 in term 0.
I20260501 02:02:34.372247 20964 leader_election.cc:304] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, d794bce3a2944d5697194d1d79613baf; no voters:
I20260501 02:02:34.372486 21200 raft_consensus.cc:2804] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:34.372550 21200 raft_consensus.cc:493] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:34.372583 21200 raft_consensus.cc:3060] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.373746 21200 raft_consensus.cc:515] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.374169 21200 leader_election.cc:290] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [CANDIDATE]: Term 1 election: Requested vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409)
I20260501 02:02:34.374505 20891 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bed442a93bdb45949ebea110e3faeb5e" candidate_uuid: "38670c6b486c42e9866b3ca9289a48c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d794bce3a2944d5697194d1d79613baf"
I20260501 02:02:34.374615 20891 raft_consensus.cc:3060] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.375594 20891 raft_consensus.cc:2468] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38670c6b486c42e9866b3ca9289a48c1 in term 1.
I20260501 02:02:34.375818 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bed442a93bdb45949ebea110e3faeb5e" candidate_uuid: "38670c6b486c42e9866b3ca9289a48c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
I20260501 02:02:34.375952 20746 raft_consensus.cc:3060] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.376965 20746 raft_consensus.cc:2468] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38670c6b486c42e9866b3ca9289a48c1 in term 1.
I20260501 02:02:34.377312 20967 leader_election.cc:304] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38670c6b486c42e9866b3ca9289a48c1, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:34.377529 21200 raft_consensus.cc:2804] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:34.377756 21200 raft_consensus.cc:697] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 1 LEADER]: Becoming Leader. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Running, Role: LEADER
I20260501 02:02:34.378039 21200 consensus_queue.cc:237] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.380285 20589 catalog_manager.cc:5671] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195). New cstate: current_term: 1 leader_uuid: "38670c6b486c42e9866b3ca9289a48c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } health_report { overall_health: HEALTHY } } }
I20260501 02:02:34.444553 21208 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.441290 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.18.96.195:43197)
0501 02:02:34.441573 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.441576 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.441586 (+ 10us) server_negotiation.cc:408] Connection header received
0501 02:02:34.441649 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.441651 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.441698 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.441818 (+ 120us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:34.442256 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.442816 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.443553 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.443738 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.443788 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.444140 (+ 352us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.444270 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.444345 (+ 75us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.444385 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
I20260501 02:02:34.453457 21211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.450836 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.18.96.195:53763)
0501 02:02:34.451155 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:34.451168 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:34.451249 (+ 81us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:34.451452 (+ 203us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:34.451455 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:34.451461 (+ 6us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:34.451654 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.451658 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.452310 (+ 652us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.452312 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:34.452716 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.452722 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.452803 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.452830 (+ 27us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.453230 (+ 400us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.453236 (+ 6us) client_negotiation.cc:770] Sending connection context
0501 02:02:34.453302 (+ 66us) client_negotiation.cc:241] Negotiation successful
0501 02:02:34.453350 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":126,"threads_started":1}
I20260501 02:02:34.453456 21203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.450954 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:53763 (local address 127.18.96.193:32801)
0501 02:02:34.451072 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.451075 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.451230 (+ 155us) server_negotiation.cc:408] Connection header received
0501 02:02:34.451312 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.451315 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.451367 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.451432 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:34.451770 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.452215 (+ 445us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.452807 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.452923 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.452973 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.453055 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.453181 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.453326 (+ 145us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.453364 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260501 02:02:34.459393 20746 raft_consensus.cc:1275] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Refusing update from remote peer 38670c6b486c42e9866b3ca9289a48c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:34.459745 20891 raft_consensus.cc:1275] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Refusing update from remote peer 38670c6b486c42e9866b3ca9289a48c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:34.459841 21205 consensus_queue.cc:1048] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:34.460162 21200 consensus_queue.cc:1048] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:34.464337 21162 mvcc.cc:204] Tried to move back new op lower bound from 7281053509461528576 to 7281053509134880768. Current Snapshot: MvccSnapshot[applied={T|T < 7281053509461528576}]
I20260501 02:02:34.476013 21216 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.473152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43197 (local address 127.18.96.193:48481)
0501 02:02:34.473467 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:34.473485 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:34.473586 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:34.473833 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:34.473836 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:34.473844 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:02:34.474157 (+ 313us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.474167 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.474840 (+ 673us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.474844 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:34.475248 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:34.475254 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.475378 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.475407 (+ 29us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.475745 (+ 338us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.475753 (+ 8us) client_negotiation.cc:770] Sending connection context
0501 02:02:34.475839 (+ 86us) client_negotiation.cc:241] Negotiation successful
0501 02:02:34.475899 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":148,"threads_started":1}
I20260501 02:02:34.476006 21208 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.473253 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:48481 (local address 127.18.96.195:43197)
0501 02:02:34.473359 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.473364 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.473560 (+ 196us) server_negotiation.cc:408] Connection header received
0501 02:02:34.473687 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.473690 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.473729 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.473820 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:34.474259 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.474725 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.475361 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.475557 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.475593 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.475649 (+ 56us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.475737 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.475851 (+ 114us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.475889 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260501 02:02:34.497891 20589 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:34.498129 20589 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:34.500097 20589 catalog_manager.cc:5958] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Sending DeleteTablet for 3 replicas of tablet bed442a93bdb45949ebea110e3faeb5e
I20260501 02:02:34.500778 21013 tablet_service.cc:1558] Processing DeleteTablet for tablet bed442a93bdb45949ebea110e3faeb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:60970
I20260501 02:02:34.500797 20871 tablet_service.cc:1558] Processing DeleteTablet for tablet bed442a93bdb45949ebea110e3faeb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:50372
I20260501 02:02:34.500954 21184 tablet_replica.cc:333] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: stopping tablet replica
I20260501 02:02:34.500974 21185 tablet_replica.cc:333] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: stopping tablet replica
I20260501 02:02:34.501075 21184 raft_consensus.cc:2243] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:34.501107 21185 raft_consensus.cc:2243] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:34.501192 21184 raft_consensus.cc:2272] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:34.501259 20719 tablet_service.cc:1558] Processing DeleteTablet for tablet bed442a93bdb45949ebea110e3faeb5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:34 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38948
I20260501 02:02:34.501376 21185 raft_consensus.cc:2272] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:34.501372 20589 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 02:02:34.501470 21183 tablet_replica.cc:333] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: stopping tablet replica
I20260501 02:02:34.501566 21183 raft_consensus.cc:2243] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:34.501652 21183 raft_consensus.cc:2272] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260501 02:02:34.501811 20589 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:34.502552 21185 ts_tablet_manager.cc:1916] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.502640 21184 ts_tablet_manager.cc:1916] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.504879 21184 ts_tablet_manager.cc:1929] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:34.504900 21185 ts_tablet_manager.cc:1929] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:34.504973 21185 log.cc:1199] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/bed442a93bdb45949ebea110e3faeb5e
I20260501 02:02:34.504979 21184 log.cc:1199] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/bed442a93bdb45949ebea110e3faeb5e
I20260501 02:02:34.505209 21184 ts_tablet_manager.cc:1950] T bed442a93bdb45949ebea110e3faeb5e P d794bce3a2944d5697194d1d79613baf: Deleting consensus metadata
I20260501 02:02:34.505244 21185 ts_tablet_manager.cc:1950] T bed442a93bdb45949ebea110e3faeb5e P 38670c6b486c42e9866b3ca9289a48c1: Deleting consensus metadata
I20260501 02:02:34.505793 20574 catalog_manager.cc:5002] TS d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409): tablet bed442a93bdb45949ebea110e3faeb5e (table test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]) successfully deleted
I20260501 02:02:34.505878 21183 ts_tablet_manager.cc:1916] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:34.506083 20574 catalog_manager.cc:5002] TS 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197): tablet bed442a93bdb45949ebea110e3faeb5e (table test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]) successfully deleted
I20260501 02:02:34.508098 21183 ts_tablet_manager.cc:1929] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:02:34.508181 21183 log.cc:1199] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/bed442a93bdb45949ebea110e3faeb5e
I20260501 02:02:34.508440 21183 ts_tablet_manager.cc:1950] T bed442a93bdb45949ebea110e3faeb5e P f5cd416d8d7041bcb35b0b0436fdf1f5: Deleting consensus metadata
I20260501 02:02:34.509068 20577 catalog_manager.cc:5002] TS f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801): tablet bed442a93bdb45949ebea110e3faeb5e (table test-table [id=a2708247f90b4f0eaaaa2c39a9b20da9]) successfully deleted
I20260501 02:02:34.509752 20719 tablet_service.cc:1511] Processing CreateTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 (DEFAULT_TABLE table=test-table [id=fdfbe99b94d3428a82494f4ca8a02def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.510036 20719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1c56b345d4638bf241917f9fc4e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.510353 20871 tablet_service.cc:1511] Processing CreateTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 (DEFAULT_TABLE table=test-table [id=fdfbe99b94d3428a82494f4ca8a02def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.510562 20871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1c56b345d4638bf241917f9fc4e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.511547 21194 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: Bootstrap starting.
I20260501 02:02:34.512331 21194 tablet_bootstrap.cc:654] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.512547 21195 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Bootstrap starting.
I20260501 02:02:34.512856 21013 tablet_service.cc:1511] Processing CreateTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 (DEFAULT_TABLE table=test-table [id=fdfbe99b94d3428a82494f4ca8a02def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:34.513067 21013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3b1c56b345d4638bf241917f9fc4e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:34.513217 21195 tablet_bootstrap.cc:654] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.514164 21195 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: No bootstrap required, opened a new log
I20260501 02:02:34.514170 21194 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: No bootstrap required, opened a new log
I20260501 02:02:34.514230 21194 ts_tablet_manager.cc:1403] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260501 02:02:34.514230 21195 ts_tablet_manager.cc:1403] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:34.514570 21195 raft_consensus.cc:359] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.514570 21194 raft_consensus.cc:359] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.514657 21194 raft_consensus.cc:385] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.514657 21195 raft_consensus.cc:385] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.514689 21194 raft_consensus.cc:740] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cd416d8d7041bcb35b0b0436fdf1f5, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.514689 21195 raft_consensus.cc:740] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.514783 21195 consensus_queue.cc:260] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.514791 21194 consensus_queue.cc:260] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.514984 21196 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Bootstrap starting.
I20260501 02:02:34.515004 21194 ts_tablet_manager.cc:1434] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:34.515004 21195 ts_tablet_manager.cc:1434] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:34.515864 21196 tablet_bootstrap.cc:654] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:34.517457 21196 tablet_bootstrap.cc:492] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: No bootstrap required, opened a new log
I20260501 02:02:34.517542 21196 ts_tablet_manager.cc:1403] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260501 02:02:34.517997 21196 raft_consensus.cc:359] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.518100 21196 raft_consensus.cc:385] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:34.518132 21196 raft_consensus.cc:740] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38670c6b486c42e9866b3ca9289a48c1, State: Initialized, Role: FOLLOWER
I20260501 02:02:34.518273 21196 consensus_queue.cc:260] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.518518 21196 ts_tablet_manager.cc:1434] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:02:34.572196 21112 raft_consensus.cc:493] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:34.572345 21112 raft_consensus.cc:515] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.572855 21112 leader_election.cc:290] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:34.573251 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b1c56b345d4638bf241917f9fc4e92" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" is_pre_election: true
I20260501 02:02:34.573289 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b1c56b345d4638bf241917f9fc4e92" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1" is_pre_election: true
I20260501 02:02:34.573413 21033 raft_consensus.cc:2468] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:34.573421 20746 raft_consensus.cc:2468] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 0.
I20260501 02:02:34.573724 20825 leader_election.cc:304] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d794bce3a2944d5697194d1d79613baf, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:34.573879 21112 raft_consensus.cc:2804] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:34.573937 21112 raft_consensus.cc:493] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:34.574034 21112 raft_consensus.cc:3060] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.575064 21112 raft_consensus.cc:515] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.575459 21112 leader_election.cc:290] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Requested vote from peers f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801), 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197)
I20260501 02:02:34.575778 20746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b1c56b345d4638bf241917f9fc4e92" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5"
I20260501 02:02:34.575851 21033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3b1c56b345d4638bf241917f9fc4e92" candidate_uuid: "d794bce3a2944d5697194d1d79613baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38670c6b486c42e9866b3ca9289a48c1"
I20260501 02:02:34.575891 20746 raft_consensus.cc:3060] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.575958 21033 raft_consensus.cc:3060] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:34.576949 21033 raft_consensus.cc:2468] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:34.576949 20746 raft_consensus.cc:2468] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d794bce3a2944d5697194d1d79613baf in term 1.
I20260501 02:02:34.577309 20825 leader_election.cc:304] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d794bce3a2944d5697194d1d79613baf, f5cd416d8d7041bcb35b0b0436fdf1f5; no voters:
I20260501 02:02:34.577459 21112 raft_consensus.cc:2804] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:34.577608 21112 raft_consensus.cc:697] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Becoming Leader. State: Replica: d794bce3a2944d5697194d1d79613baf, State: Running, Role: LEADER
I20260501 02:02:34.577746 21112 consensus_queue.cc:237] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } }
I20260501 02:02:34.579478 20588 catalog_manager.cc:5671] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf reported cstate change: term changed from 0 to 1, leader changed from <none> to d794bce3a2944d5697194d1d79613baf (127.18.96.194). New cstate: current_term: 1 leader_uuid: "d794bce3a2944d5697194d1d79613baf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d794bce3a2944d5697194d1d79613baf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
I20260501 02:02:34.603735 21204 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:34.600661 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.18.96.194:35409)
0501 02:02:34.600783 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:34.600787 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:34.600920 (+ 133us) server_negotiation.cc:408] Connection header received
0501 02:02:34.601049 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:34.601052 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:34.601097 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:34.601160 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:34.601599 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.602386 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:34.603054 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:34.603267 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:34.603311 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:34.603386 (+ 75us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:34.603465 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:34.603573 (+ 108us) server_negotiation.cc:300] Negotiation successful
0501 02:02:34.603614 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260501 02:02:34.609299 20746 raft_consensus.cc:1275] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5 [term 1 FOLLOWER]: Refusing update from remote peer d794bce3a2944d5697194d1d79613baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:34.609414 21033 raft_consensus.cc:1275] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Refusing update from remote peer d794bce3a2944d5697194d1d79613baf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:34.609817 21222 consensus_queue.cc:1048] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cd416d8d7041bcb35b0b0436fdf1f5" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:34.610034 21112 consensus_queue.cc:1048] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38670c6b486c42e9866b3ca9289a48c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260501 02:02:34.622403 21165 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260501 02:02:36.171861 20588 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:36.171999 20588 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:36.174501 20588 catalog_manager.cc:5958] T 00000000000000000000000000000000 P afaa50d2fcce4442b156ce9ac1e76e39: Sending DeleteTablet for 3 replicas of tablet b3b1c56b345d4638bf241917f9fc4e92
I20260501 02:02:36.175014 20719 tablet_service.cc:1558] Processing DeleteTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:36 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38948
I20260501 02:02:36.175207 21013 tablet_service.cc:1558] Processing DeleteTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:36 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:60970
I20260501 02:02:36.175346 21251 tablet_replica.cc:333] T b3b1c56b345d4638bf241917f9fc4e92 P f5cd416d8d7041bcb35b0b0436fdf1f5: stopping tablet replica
I20260501 02:02:36.175423 20871 tablet_service.cc:1558] Processing DeleteTablet for tablet b3b1c56b345d4638bf241917f9fc4e92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:36 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:50372
I20260501 02:02:36.175446 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20664
I20260501 02:02:36.177412 21253 tablet_replica.cc:333] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: stopping tablet replica
I20260501 02:02:36.177560 21253 raft_consensus.cc:2243] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:36.177767 21253 raft_consensus.cc:2272] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:36.178053 21252 tablet_replica.cc:333] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: stopping tablet replica
I20260501 02:02:36.178292 21252 raft_consensus.cc:2243] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:36.178442 21252 raft_consensus.cc:2272] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:36.178509 21253 ts_tablet_manager.cc:1916] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:36.179111 21252 ts_tablet_manager.cc:1916] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:36.180711 21253 ts_tablet_manager.cc:1929] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260501 02:02:36.180810 21253 log.cc:1199] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/b3b1c56b345d4638bf241917f9fc4e92
I20260501 02:02:36.181154 21253 ts_tablet_manager.cc:1950] T b3b1c56b345d4638bf241917f9fc4e92 P d794bce3a2944d5697194d1d79613baf: Deleting consensus metadata
I20260501 02:02:36.181840 20574 catalog_manager.cc:5002] TS d794bce3a2944d5697194d1d79613baf (127.18.96.194:35409): tablet b3b1c56b345d4638bf241917f9fc4e92 (table test-table [id=fdfbe99b94d3428a82494f4ca8a02def]) successfully deleted
I20260501 02:02:36.182219 21252 ts_tablet_manager.cc:1929] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260501 02:02:36.182300 21252 log.cc:1199] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/b3b1c56b345d4638bf241917f9fc4e92
I20260501 02:02:36.182588 21252 ts_tablet_manager.cc:1950] T b3b1c56b345d4638bf241917f9fc4e92 P 38670c6b486c42e9866b3ca9289a48c1: Deleting consensus metadata
I20260501 02:02:36.183326 20574 catalog_manager.cc:5002] TS 38670c6b486c42e9866b3ca9289a48c1 (127.18.96.195:43197): tablet b3b1c56b345d4638bf241917f9fc4e92 (table test-table [id=fdfbe99b94d3428a82494f4ca8a02def]) successfully deleted
W20260501 02:02:36.185083 20577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:32801 (error 108)
I20260501 02:02:36.185258 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20806
I20260501 02:02:36.186923 21255 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:36.186425 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:32801 (local address 127.0.0.1:38978)
0501 02:02:36.186721 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:36.186790 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:32801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
W20260501 02:02:36.187072 20577 catalog_manager.cc:4729] TS f5cd416d8d7041bcb35b0b0436fdf1f5 (127.18.96.193:32801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3b1c56b345d4638bf241917f9fc4e92: Network error: Client connection negotiation failed: client connection to 127.18.96.193:32801: connect: Connection refused (error 111)
I20260501 02:02:36.192746 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20948
I20260501 02:02:36.199419 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 20558
2026-05-01T02:02:36Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10080 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:36 dist-test-slave-2j90 krb5kdc[21258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:36 dist-test-slave-2j90 krb5kdc[21258](info): set up 2 sockets
May 01 02:02:36 dist-test-slave-2j90 krb5kdc[21258](info): commencing operation
krb5kdc: starting...
W20260501 02:02:38.198421 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.976s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:38.329550 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33969
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:38.427222 21274 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:38.427409 21274 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:38.427457 21274 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:38.430688 21274 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:38.430735 21274 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:38.430756 21274 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:38.430773 21274 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:38.430787 21274 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:38.434720 21274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:38.435568 21274 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:38.436264 21274 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:38.441634 21280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:38.441684 21279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:38.441797 21282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:38.441699 21274 server_base.cc:1061] running on GCE node
I20260501 02:02:38.442363 21274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:38.443145 21274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:38.444294 21274 hybrid_clock.cc:648] HybridClock initialized: now 1777600958444276 us; error 30 us; skew 500 ppm
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:38.446806 21274 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:38.447788 21274 webserver.cc:492] Webserver started at http://127.18.96.254:39287/ using document root <none> and password file <none>
I20260501 02:02:38.448244 21274 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:38.448289 21274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:38.448433 21274 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:38.450007 21274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "6642553f9a62423b9259843b656dafb7"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "2a410e546f726bccb0e6c0924725b46b"
server_key_iv: "3a696f18ff70017bc74dae4ee55a2818"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.450431 21274 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "6642553f9a62423b9259843b656dafb7"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "2a410e546f726bccb0e6c0924725b46b"
server_key_iv: "3a696f18ff70017bc74dae4ee55a2818"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.453212 21274 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:02:38.455101 21289 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.455961 21274 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:38.456095 21274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "6642553f9a62423b9259843b656dafb7"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "2a410e546f726bccb0e6c0924725b46b"
server_key_iv: "3a696f18ff70017bc74dae4ee55a2818"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.456189 21274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:38.463781 21274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:38.466306 21274 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:38.466490 21274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:38.473250 21274 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:33969
I20260501 02:02:38.473263 21341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:33969 every 8 connection(s)
I20260501 02:02:38.474180 21274 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:38.476552 21342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:38.481310 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7: Bootstrap starting.
I20260501 02:02:38.483378 21342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:38.483958 21342 log.cc:826] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:38.484386 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21274
I20260501 02:02:38.484475 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:38.484746 18819 external_mini_cluster.cc:1468] Setting key 006b247e455841e69acceab86d0f9e41
I20260501 02:02:38.486021 21342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7: No bootstrap required, opened a new log
I20260501 02:02:38.488719 21342 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } }
I20260501 02:02:38.488902 21342 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:38.488983 21342 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6642553f9a62423b9259843b656dafb7, State: Initialized, Role: FOLLOWER
I20260501 02:02:38.489391 21342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } }
I20260501 02:02:38.489549 21342 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:38.489611 21342 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:38.489689 21342 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:38.490758 21342 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } }
I20260501 02:02:38.491127 21342 leader_election.cc:304] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6642553f9a62423b9259843b656dafb7; no voters:
I20260501 02:02:38.491485 21342 leader_election.cc:290] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:38.491608 21347 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:38.491822 21347 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [term 1 LEADER]: Becoming Leader. State: Replica: 6642553f9a62423b9259843b656dafb7, State: Running, Role: LEADER
I20260501 02:02:38.492118 21347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } }
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:38.492558 21342 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:38.493901 21348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6642553f9a62423b9259843b656dafb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } } }
I20260501 02:02:38.494307 21348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:38.494623 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6642553f9a62423b9259843b656dafb7. Latest consensus state: current_term: 1 leader_uuid: "6642553f9a62423b9259843b656dafb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6642553f9a62423b9259843b656dafb7" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33969 } } }
I20260501 02:02:38.494750 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:38.495046 21356 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:38.497581 21346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:38.485801 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.18.96.254:33969)
0501 02:02:38.486197 (+ 396us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:38.486206 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:38.486271 (+ 65us) server_negotiation.cc:408] Connection header received
0501 02:02:38.486771 (+ 500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:38.486789 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:38.487053 (+ 264us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:38.487299 (+ 246us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:38.488328 (+ 1029us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.489326 (+ 998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:38.490288 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.490570 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:38.493044 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:38.493065 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:38.493082 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:38.493110 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:38.495257 (+ 2147us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.495778 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.495783 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.495789 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.495864 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.496118 (+ 254us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.496122 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.496124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.496409 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:38.496608 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:38.496849 (+ 241us) server_negotiation.cc:300] Negotiation successful
0501 02:02:38.497072 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":151,"threads_started":1}
I20260501 02:02:38.497721 21356 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:38.503432 21356 catalog_manager.cc:1357] Generated new cluster ID: e6028ed544594c87b4414d30c8211bc1
I20260501 02:02:38.503495 21356 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:38.514539 21356 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:38.515497 21356 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:38.531522 21356 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6642553f9a62423b9259843b656dafb7: Generated new TSK 0
I20260501 02:02:38.532089 21356 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:38.587402 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:38.684995 21370 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:38.685205 21370 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:38.685295 21370 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:38.688364 21370 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:38.688444 21370 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:38.688544 21370 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:38.692538 21370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:38.693527 21370 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:38.694342 21370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:38.700261 21376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:38.700251 21378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:38.700253 21375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:38.700629 21370 server_base.cc:1061] running on GCE node
I20260501 02:02:38.700980 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:38.701472 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:38.702618 21370 hybrid_clock.cc:648] HybridClock initialized: now 1777600958702592 us; error 38 us; skew 500 ppm
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:38.705035 21370 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:38.706061 21370 webserver.cc:492] Webserver started at http://127.18.96.193:34279/ using document root <none> and password file <none>
I20260501 02:02:38.706566 21370 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:38.706638 21370 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:38.706841 21370 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:38.708313 21370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "c31afce2af154c0580cc1c0167e22af3"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "acf826839f02214d88194aad4e7df1ca"
server_key_iv: "c4a36c574db1347bdad35b2defafc58e"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.708770 21370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "c31afce2af154c0580cc1c0167e22af3"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "acf826839f02214d88194aad4e7df1ca"
server_key_iv: "c4a36c574db1347bdad35b2defafc58e"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.711809 21370 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:38.713727 21385 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.714649 21370 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:38.714775 21370 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "c31afce2af154c0580cc1c0167e22af3"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "acf826839f02214d88194aad4e7df1ca"
server_key_iv: "c4a36c574db1347bdad35b2defafc58e"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.714867 21370 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:38.726181 21370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:38.728230 21370 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:38.728410 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:38.728885 21370 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:38.729676 21370 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:38.729745 21370 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.729810 21370 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:38.729857 21370 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.738627 21370 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:45229
I20260501 02:02:38.738668 21498 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:45229 every 8 connection(s)
I20260501 02:02:38.739476 21370 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:38.742616 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21370
I20260501 02:02:38.742746 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:38.743034 18819 external_mini_cluster.cc:1468] Setting key 86d20ca9b5280b67a23360876457dbe0
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:38.751999 21346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:38.741074 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:49087 (local address 127.18.96.254:33969)
0501 02:02:38.741216 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:38.741220 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:38.741781 (+ 561us) server_negotiation.cc:408] Connection header received
0501 02:02:38.742762 (+ 981us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:38.742768 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:38.742854 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:38.742931 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:38.744847 (+ 1916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.745349 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:38.746230 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.746460 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:38.748638 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:38.748662 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:38.748664 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:38.748687 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:38.750170 (+ 1483us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.750631 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.750635 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.750636 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.750695 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.750967 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.750971 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.750973 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.751111 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:38.751212 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:38.751689 (+ 477us) server_negotiation.cc:300] Negotiation successful
0501 02:02:38.751843 (+ 154us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:02:38.752527 21501 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:38.741283 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33969 (local address 127.18.96.193:49087)
0501 02:02:38.741646 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:38.741674 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:38.742524 (+ 850us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:38.743116 (+ 592us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:38.743126 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:38.743615 (+ 489us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:38.744674 (+ 1059us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:38.744692 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.745478 (+ 786us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:38.745485 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:38.746119 (+ 634us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:38.746127 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.746271 (+ 144us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:38.746834 (+ 563us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:38.746862 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:38.748497 (+ 1635us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:38.750286 (+ 1789us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:38.750299 (+ 13us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:38.750310 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:38.750528 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:38.750795 (+ 267us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:38.750798 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:38.750800 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:38.750878 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:38.751213 (+ 335us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:38.751219 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:38.751443 (+ 224us) client_negotiation.cc:770] Sending connection context
0501 02:02:38.751645 (+ 202us) client_negotiation.cc:241] Negotiation successful
0501 02:02:38.751871 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260501 02:02:38.753578 21499 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33969
I20260501 02:02:38.753798 21499 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:38.754313 21499 heartbeater.cc:507] Master 127.18.96.254:33969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:02:38.755827 21306 ts_manager.cc:194] Registered new tserver with Master: c31afce2af154c0580cc1c0167e22af3 (127.18.96.193:45229)
I20260501 02:02:38.757512 21306 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:49087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:38.789333 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:38.885923 21506 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:38.886175 21506 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:38.886262 21506 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:38.889467 21506 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:38.889562 21506 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:38.889672 21506 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:38.893656 21506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:38.894649 21506 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:38.895504 21506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:38.901692 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:38.901736 21506 server_base.cc:1061] running on GCE node
W20260501 02:02:38.901696 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:38.901805 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:38.902235 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:38.902720 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:38.903867 21506 hybrid_clock.cc:648] HybridClock initialized: now 1777600958903841 us; error 37 us; skew 500 ppm
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:38.906392 21506 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:38.907303 21506 webserver.cc:492] Webserver started at http://127.18.96.194:46077/ using document root <none> and password file <none>
I20260501 02:02:38.907788 21506 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:38.907853 21506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:38.908032 21506 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:38.909511 21506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "0f55bf27fef84cb183ceed7b868b3384"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "d753d2203814fdc99d371c785f1f6278"
server_key_iv: "51c5aa235561db30840b6a23edf366e6"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.910033 21506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "0f55bf27fef84cb183ceed7b868b3384"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "d753d2203814fdc99d371c785f1f6278"
server_key_iv: "51c5aa235561db30840b6a23edf366e6"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.913039 21506 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:02:38.915038 21521 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.915928 21506 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:38.916047 21506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "0f55bf27fef84cb183ceed7b868b3384"
format_stamp: "Formatted at 2026-05-01 02:02:38 on dist-test-slave-2j90"
server_key: "d753d2203814fdc99d371c785f1f6278"
server_key_iv: "51c5aa235561db30840b6a23edf366e6"
server_key_version: "encryptionkey@0"
I20260501 02:02:38.916137 21506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:38.929728 21506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:38.932073 21506 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:38.932252 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:38.932765 21506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:38.933590 21506 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:38.933665 21506 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.933745 21506 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:38.933784 21506 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:38.942725 21506 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:33853
I20260501 02:02:38.942795 21634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:33853 every 8 connection(s)
I20260501 02:02:38.943603 21506 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:38.943984 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21506
I20260501 02:02:38.944113 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:38.944437 18819 external_mini_cluster.cc:1468] Setting key fd79f80a123ed7e3b71d365275354852
May 01 02:02:38 dist-test-slave-2j90 krb5kdc[21258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:38.956017 21346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:38.945233 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40977 (local address 127.18.96.254:33969)
0501 02:02:38.945381 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:38.945386 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:38.946348 (+ 962us) server_negotiation.cc:408] Connection header received
0501 02:02:38.947556 (+ 1208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:38.947559 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:38.947605 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:38.947675 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:38.949185 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.949748 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:38.950430 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.950613 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:38.952712 (+ 2099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:38.952736 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:38.952739 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:38.952763 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:38.954203 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.954699 (+ 496us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.954703 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.954704 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.954755 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:38.955007 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:38.955009 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:38.955010 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:38.955130 (+ 120us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:38.955201 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:38.955774 (+ 573us) server_negotiation.cc:300] Negotiation successful
0501 02:02:38.955869 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:38.956643 21637 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:38.945582 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33969 (local address 127.18.96.194:40977)
0501 02:02:38.946189 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:38.946228 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:38.947331 (+ 1103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:38.947879 (+ 548us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:38.947893 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:38.948255 (+ 362us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:38.949047 (+ 792us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:38.949059 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.949873 (+ 814us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:38.949879 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:38.950314 (+ 435us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:38.950321 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:38.950517 (+ 196us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:38.951148 (+ 631us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:38.951164 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:38.952574 (+ 1410us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:38.954334 (+ 1760us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:38.954339 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:38.954349 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:38.954557 (+ 208us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:38.954851 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:38.954854 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:38.954855 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:38.954932 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:38.955284 (+ 352us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:38.955290 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:38.955492 (+ 202us) client_negotiation.cc:770] Sending connection context
0501 02:02:38.955723 (+ 231us) client_negotiation.cc:241] Negotiation successful
0501 02:02:38.955895 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":173,"threads_started":1}
I20260501 02:02:38.957635 21635 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33969
I20260501 02:02:38.957918 21635 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:38.958524 21635 heartbeater.cc:507] Master 127.18.96.254:33969 requested a full tablet report, sending...
I20260501 02:02:38.959666 21306 ts_manager.cc:194] Registered new tserver with Master: 0f55bf27fef84cb183ceed7b868b3384 (127.18.96.194:33853)
I20260501 02:02:38.960466 21306 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:40977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:38.993266 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:39.093835 21642 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:39.094091 21642 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:39.094184 21642 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:39.097402 21642 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:39.097502 21642 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:39.097615 21642 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:39.101918 21642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:39.102977 21642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:39.103760 21642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:39.109706 21647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:39.109804 21648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:39.109706 21650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:39.109875 21642 server_base.cc:1061] running on GCE node
I20260501 02:02:39.110352 21642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:39.110801 21642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:39.111941 21642 hybrid_clock.cc:648] HybridClock initialized: now 1777600959111926 us; error 27 us; skew 500 ppm
May 01 02:02:39 dist-test-slave-2j90 krb5kdc[21258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600959, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:39.114446 21642 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:39.115362 21642 webserver.cc:492] Webserver started at http://127.18.96.195:33373/ using document root <none> and password file <none>
I20260501 02:02:39.115861 21642 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:39.115907 21642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:39.116086 21642 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:39.117578 21642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "2d8a1fb34d8146eca6783f247e3d26ee"
format_stamp: "Formatted at 2026-05-01 02:02:39 on dist-test-slave-2j90"
server_key: "4e19cb010b3cde8696aa89219aff8320"
server_key_iv: "703db078b748e30dac50dd8021c14e69"
server_key_version: "encryptionkey@0"
I20260501 02:02:39.118132 21642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "2d8a1fb34d8146eca6783f247e3d26ee"
format_stamp: "Formatted at 2026-05-01 02:02:39 on dist-test-slave-2j90"
server_key: "4e19cb010b3cde8696aa89219aff8320"
server_key_iv: "703db078b748e30dac50dd8021c14e69"
server_key_version: "encryptionkey@0"
I20260501 02:02:39.121253 21642 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:39.123327 21657 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:39.124228 21642 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:39.124344 21642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "2d8a1fb34d8146eca6783f247e3d26ee"
format_stamp: "Formatted at 2026-05-01 02:02:39 on dist-test-slave-2j90"
server_key: "4e19cb010b3cde8696aa89219aff8320"
server_key_iv: "703db078b748e30dac50dd8021c14e69"
server_key_version: "encryptionkey@0"
I20260501 02:02:39.124433 21642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:39.139863 21642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:39.142037 21642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:39.142225 21642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:39.142735 21642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:39.143537 21642 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:39.143581 21642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:39.143643 21642 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:39.143680 21642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:39.153019 21642 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:41737
I20260501 02:02:39.153044 21770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:41737 every 8 connection(s)
I20260501 02:02:39.153851 21642 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:39.158393 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21642
I20260501 02:02:39.158483 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:39.158715 18819 external_mini_cluster.cc:1468] Setting key 6433e12b2116f4acbc80a30bb0d5a90a
May 01 02:02:39 dist-test-slave-2j90 krb5kdc[21258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600959, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:39.165400 21346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:39.155428 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:52179 (local address 127.18.96.254:33969)
0501 02:02:39.155573 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:39.155576 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:39.156152 (+ 576us) server_negotiation.cc:408] Connection header received
0501 02:02:39.156859 (+ 707us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:39.156862 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:39.156905 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:39.156974 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:39.158618 (+ 1644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:39.159222 (+ 604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:39.160037 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:39.160192 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:39.162336 (+ 2144us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:39.162356 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:39.162358 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:39.162383 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:39.163625 (+ 1242us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:39.164084 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:39.164088 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:39.164090 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:39.164139 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:39.164432 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:39.164435 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:39.164437 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:39.164584 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:39.164690 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:39.165138 (+ 448us) server_negotiation.cc:300] Negotiation successful
0501 02:02:39.165246 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:02:39.166088 21773 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:39.155660 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33969 (local address 127.18.96.195:52179)
0501 02:02:39.156017 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:39.156059 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:39.156708 (+ 649us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:39.157121 (+ 413us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:39.157128 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:39.157505 (+ 377us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:39.158399 (+ 894us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:39.158411 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:39.159375 (+ 964us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:39.159378 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:39.159926 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:39.159935 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:39.160115 (+ 180us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:39.160664 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:39.160689 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:39.162188 (+ 1499us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:39.163747 (+ 1559us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:39.163753 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:39.163764 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:39.163981 (+ 217us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:39.164243 (+ 262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:39.164246 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:39.164247 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:39.164337 (+ 90us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:39.164690 (+ 353us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:39.164696 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:39.164900 (+ 204us) client_negotiation.cc:770] Sending connection context
0501 02:02:39.165104 (+ 204us) client_negotiation.cc:241] Negotiation successful
0501 02:02:39.165318 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
I20260501 02:02:39.167062 21771 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33969
I20260501 02:02:39.167335 21771 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:39.167784 21771 heartbeater.cc:507] Master 127.18.96.254:33969 requested a full tablet report, sending...
I20260501 02:02:39.168722 21306 ts_manager.cc:194] Registered new tserver with Master: 2d8a1fb34d8146eca6783f247e3d26ee (127.18.96.195:41737)
I20260501 02:02:39.169229 21306 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:52179
I20260501 02:02:39.172328 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260501 02:02:39.183758 21346 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:39.182374 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.18.96.254:33969)
0501 02:02:39.182538 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:39.182543 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:39.182701 (+ 158us) server_negotiation.cc:408] Connection header received
0501 02:02:39.182852 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:39.182862 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:39.182903 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:39.182988 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:39.183633 (+ 645us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52304: BlockingRecv error: recv got EOF from 127.0.0.1:52304 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:02:39.185204 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21370
I20260501 02:02:39.190939 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21506
I20260501 02:02:39.196666 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21642
I20260501 02:02:39.202237 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21274
2026-05-01T02:02:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3001 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:39 dist-test-slave-2j90 krb5kdc[21783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:39 dist-test-slave-2j90 krb5kdc[21783](info): set up 2 sockets
May 01 02:02:39 dist-test-slave-2j90 krb5kdc[21783](info): commencing operation
krb5kdc: starting...
W20260501 02:02:41.198421 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:41.330894 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46209
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:41.428243 21799 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:41.428490 21799 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:41.428568 21799 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:41.431774 21799 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:41.431862 21799 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:41.431903 21799 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:41.431979 21799 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:41.432018 21799 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:41.436060 21799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:46209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:41.437075 21799 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:41.437863 21799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:41.443316 21805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.443328 21807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.443341 21804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:41.443532 21799 server_base.cc:1061] running on GCE node
I20260501 02:02:41.444092 21799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:41.444909 21799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:41.446017 21799 hybrid_clock.cc:648] HybridClock initialized: now 1777600961446000 us; error 29 us; skew 500 ppm
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:41.448657 21799 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:41.449661 21799 webserver.cc:492] Webserver started at http://127.18.96.254:46231/ using document root <none> and password file <none>
I20260501 02:02:41.450224 21799 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:41.450273 21799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:41.450465 21799 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:41.451973 21799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "dcf4293a45494ca986da219fd73abc1e"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "b3160b9846b2624e2764a40d55df17c4"
server_key_iv: "2dafe44754d8ed1a45c0b7ecd7951746"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.452450 21799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "dcf4293a45494ca986da219fd73abc1e"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "b3160b9846b2624e2764a40d55df17c4"
server_key_iv: "2dafe44754d8ed1a45c0b7ecd7951746"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.455521 21799 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:41.457427 21814 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.458310 21799 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:41.458397 21799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "dcf4293a45494ca986da219fd73abc1e"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "b3160b9846b2624e2764a40d55df17c4"
server_key_iv: "2dafe44754d8ed1a45c0b7ecd7951746"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.458509 21799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:41.476040 21799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:41.481397 21799 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:41.481576 21799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:41.488538 21799 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:46209
I20260501 02:02:41.488554 21866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:46209 every 8 connection(s)
I20260501 02:02:41.489539 21799 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:41.492128 21867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:41.495616 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21799
I20260501 02:02:41.495711 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:41.495944 18819 external_mini_cluster.cc:1468] Setting key 993c21b26c9848640d4e8e277ff53dee
I20260501 02:02:41.497435 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e: Bootstrap starting.
I20260501 02:02:41.499617 21867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:41.500304 21867 log.cc:826] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:41.502408 21867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e: No bootstrap required, opened a new log
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:41.505522 21867 raft_consensus.cc:359] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } }
I20260501 02:02:41.505749 21867 raft_consensus.cc:385] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:41.505829 21867 raft_consensus.cc:740] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4293a45494ca986da219fd73abc1e, State: Initialized, Role: FOLLOWER
I20260501 02:02:41.506215 21867 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } }
I20260501 02:02:41.506379 21867 raft_consensus.cc:399] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:41.506460 21867 raft_consensus.cc:493] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:41.506567 21867 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:41.507360 21867 raft_consensus.cc:515] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } }
I20260501 02:02:41.507704 21867 leader_election.cc:304] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcf4293a45494ca986da219fd73abc1e; no voters:
I20260501 02:02:41.507992 21867 leader_election.cc:290] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:41.508054 21872 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:41.508261 21872 raft_consensus.cc:697] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [term 1 LEADER]: Becoming Leader. State: Replica: dcf4293a45494ca986da219fd73abc1e, State: Running, Role: LEADER
I20260501 02:02:41.508477 21870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:41.497142 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.18.96.254:46209)
0501 02:02:41.497564 (+ 422us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:41.497577 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:41.497623 (+ 46us) server_negotiation.cc:408] Connection header received
0501 02:02:41.498144 (+ 521us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:41.498159 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:41.498407 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:41.498646 (+ 239us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:41.499714 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.500456 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:41.501230 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.501474 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:41.503845 (+ 2371us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:41.503861 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:41.503872 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:41.503898 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:41.505743 (+ 1845us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.506214 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.506218 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.506222 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.506284 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.506590 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.506593 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.506595 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.506916 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:41.507115 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:41.507375 (+ 260us) server_negotiation.cc:300] Negotiation successful
0501 02:02:41.507585 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":138,"threads_started":1}
I20260501 02:02:41.508559 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } }
I20260501 02:02:41.509052 21867 sys_catalog.cc:565] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:41.509745 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcf4293a45494ca986da219fd73abc1e. Latest consensus state: current_term: 1 leader_uuid: "dcf4293a45494ca986da219fd73abc1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } } }
I20260501 02:02:41.509930 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:41.510782 21880 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:41.510782 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcf4293a45494ca986da219fd73abc1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcf4293a45494ca986da219fd73abc1e" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46209 } } }
I20260501 02:02:41.511137 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:41.514072 21880 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:41.518836 21880 catalog_manager.cc:1357] Generated new cluster ID: be841eb3d6db4783868fd8b129b5946d
I20260501 02:02:41.518899 21880 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:41.533365 21880 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:41.534435 21880 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:41.541170 21880 catalog_manager.cc:6044] T 00000000000000000000000000000000 P dcf4293a45494ca986da219fd73abc1e: Generated new TSK 0
I20260501 02:02:41.541699 21880 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:41.592634 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46209
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:41.688683 21895 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:41.688860 21895 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:41.688910 21895 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:41.692080 21895 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:41.692129 21895 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:41.692248 21895 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:41.695979 21895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:41.696915 21895 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:41.697642 21895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:41.703476 21903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.703516 21901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.703476 21900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:41.703744 21895 server_base.cc:1061] running on GCE node
I20260501 02:02:41.704123 21895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:41.704675 21895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:41.705833 21895 hybrid_clock.cc:648] HybridClock initialized: now 1777600961705815 us; error 41 us; skew 500 ppm
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:41.708288 21895 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:41.709184 21895 webserver.cc:492] Webserver started at http://127.18.96.193:34629/ using document root <none> and password file <none>
I20260501 02:02:41.709726 21895 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:41.709817 21895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:41.710064 21895 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:41.711755 21895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "b2e97f314e394b53858f6fdb18051567"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "3f5ff1df4c6874ebafac7b3aa273d96e"
server_key_iv: "a6f62a2967d24b342c907b7b3f0adf76"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.712229 21895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "b2e97f314e394b53858f6fdb18051567"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "3f5ff1df4c6874ebafac7b3aa273d96e"
server_key_iv: "a6f62a2967d24b342c907b7b3f0adf76"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.715241 21895 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:02:41.717142 21910 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.718122 21895 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:02:41.718235 21895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "b2e97f314e394b53858f6fdb18051567"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "3f5ff1df4c6874ebafac7b3aa273d96e"
server_key_iv: "a6f62a2967d24b342c907b7b3f0adf76"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.718335 21895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:41.729415 21895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:41.731608 21895 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:41.731812 21895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:41.732295 21895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:41.733095 21895 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:41.733166 21895 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.733228 21895 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:41.733280 21895 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.741458 21895 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44075
I20260501 02:02:41.741485 22023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44075 every 8 connection(s)
I20260501 02:02:41.742295 21895 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:41.747256 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 21895
I20260501 02:02:41.747365 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:41.747645 18819 external_mini_cluster.cc:1468] Setting key 1575dbf566425ec1858651108859f344
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:41.754736 21870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:41.743858 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:44541 (local address 127.18.96.254:46209)
0501 02:02:41.743997 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:41.744001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:41.744592 (+ 591us) server_negotiation.cc:408] Connection header received
0501 02:02:41.745339 (+ 747us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:41.745343 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:41.745401 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:41.745505 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:41.746576 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.747068 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:41.747850 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.748042 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:41.751048 (+ 3006us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:41.751069 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:41.751074 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:41.751108 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:41.752691 (+ 1583us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.753298 (+ 607us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.753301 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.753302 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.753361 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.753705 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.753707 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.753709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.753846 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:41.753917 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:41.754506 (+ 589us) server_negotiation.cc:300] Negotiation successful
0501 02:02:41.754603 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:02:41.755622 22026 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:41.744081 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46209 (local address 127.18.96.193:44541)
0501 02:02:41.744452 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:41.744481 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:41.745163 (+ 682us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:41.745645 (+ 482us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:41.745653 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:41.745989 (+ 336us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:41.746435 (+ 446us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:41.746445 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.747183 (+ 738us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:41.747188 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:41.747727 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:41.747743 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.747928 (+ 185us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:41.749121 (+ 1193us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:41.749138 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:41.750880 (+ 1742us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:41.752814 (+ 1934us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:41.752823 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:41.752838 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:41.753184 (+ 346us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:41.753455 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:41.753462 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:41.753466 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:41.753596 (+ 130us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:41.753973 (+ 377us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:41.753980 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:41.754254 (+ 274us) client_negotiation.cc:770] Sending connection context
0501 02:02:41.754488 (+ 234us) client_negotiation.cc:241] Negotiation successful
0501 02:02:41.754737 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:41.756788 22024 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46209
I20260501 02:02:41.757077 22024 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:41.757555 22024 heartbeater.cc:507] Master 127.18.96.254:46209 requested a full tablet report, sending...
I20260501 02:02:41.759027 21831 ts_manager.cc:194] Registered new tserver with Master: b2e97f314e394b53858f6fdb18051567 (127.18.96.193:44075)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:02:41.760229 21831 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:44541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:41.794705 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46209
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:41.892012 22031 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:41.892242 22031 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:41.892342 22031 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:41.895535 22031 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:41.895622 22031 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:41.895751 22031 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:41.899724 22031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:41.900695 22031 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:41.901503 22031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 02:02:41.907958 22031 server_base.cc:1061] running on GCE node
W20260501 02:02:41.907892 22039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.907905 22036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:41.907917 22037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:41.908459 22031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:41.908982 22031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:41.910161 22031 hybrid_clock.cc:648] HybridClock initialized: now 1777600961910143 us; error 39 us; skew 500 ppm
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:41.912753 22031 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:41.913609 22031 webserver.cc:492] Webserver started at http://127.18.96.194:36431/ using document root <none> and password file <none>
I20260501 02:02:41.914160 22031 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:41.914208 22031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:41.914362 22031 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:41.915858 22031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "cd407b677c654afe94ef663fa9b0eefe"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "cb1e0b4f8f864fe8540e08c7834fb9e4"
server_key_iv: "1c525ef092159cb64dce06bc3a656165"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.916239 22031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "cd407b677c654afe94ef663fa9b0eefe"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "cb1e0b4f8f864fe8540e08c7834fb9e4"
server_key_iv: "1c525ef092159cb64dce06bc3a656165"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.919173 22031 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:02:41.920998 22046 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.921944 22031 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:41.922091 22031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "cd407b677c654afe94ef663fa9b0eefe"
format_stamp: "Formatted at 2026-05-01 02:02:41 on dist-test-slave-2j90"
server_key: "cb1e0b4f8f864fe8540e08c7834fb9e4"
server_key_iv: "1c525ef092159cb64dce06bc3a656165"
server_key_version: "encryptionkey@0"
I20260501 02:02:41.922159 22031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:41.945016 22031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:41.947422 22031 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:41.947557 22031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:41.948014 22031 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:41.948791 22031 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:41.948834 22031 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.948871 22031 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:41.948885 22031 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:41.957544 22031 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:33483
I20260501 02:02:41.957585 22159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:33483 every 8 connection(s)
I20260501 02:02:41.958456 22031 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:41.959705 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22031
I20260501 02:02:41.959827 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:41.960103 18819 external_mini_cluster.cc:1468] Setting key e1342165a5ac65c27e2422eda96593ce
May 01 02:02:41 dist-test-slave-2j90 krb5kdc[21783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600961, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:41.970911 21870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:41.960044 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:56921 (local address 127.18.96.254:46209)
0501 02:02:41.960200 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:41.960205 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:41.960963 (+ 758us) server_negotiation.cc:408] Connection header received
0501 02:02:41.961922 (+ 959us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:41.961925 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:41.962026 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:41.962113 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:41.963629 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.964453 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:41.965080 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.965312 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:41.967693 (+ 2381us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:41.967710 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:41.967713 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:41.967743 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:41.969297 (+ 1554us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.969729 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.969732 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.969733 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.969774 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:41.970078 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:41.970080 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:41.970100 (+ 20us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:41.970244 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:41.970322 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:41.970706 (+ 384us) server_negotiation.cc:300] Negotiation successful
0501 02:02:41.970795 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:02:41.971627 22162 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:41.960342 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46209 (local address 127.18.96.194:56921)
0501 02:02:41.960836 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:41.960879 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:41.961720 (+ 841us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:41.962365 (+ 645us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:41.962378 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:41.962892 (+ 514us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:41.963473 (+ 581us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:41.963485 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.964569 (+ 1084us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:41.964572 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:41.964959 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:41.964965 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:41.965127 (+ 162us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:41.966180 (+ 1053us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:41.966197 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:41.967549 (+ 1352us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:41.969405 (+ 1856us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:41.969410 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:41.969417 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:41.969618 (+ 201us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:41.969874 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:41.969876 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:41.969877 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:41.969990 (+ 113us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:41.970340 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:41.970344 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:41.970513 (+ 169us) client_negotiation.cc:770] Sending connection context
0501 02:02:41.970673 (+ 160us) client_negotiation.cc:241] Negotiation successful
0501 02:02:41.970829 (+ 156us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:41.972707 22160 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46209
I20260501 02:02:41.972934 22160 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:41.973484 22160 heartbeater.cc:507] Master 127.18.96.254:46209 requested a full tablet report, sending...
I20260501 02:02:41.974714 21831 ts_manager.cc:194] Registered new tserver with Master: cd407b677c654afe94ef663fa9b0eefe (127.18.96.194:33483)
I20260501 02:02:41.975310 21831 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:56921
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:42.007297 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46209
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:42.104569 22167 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:42.104768 22167 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:42.104820 22167 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:42.108009 22167 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:42.108060 22167 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:42.108131 22167 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:42.112237 22167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:46209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:42.113123 22167 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:42.113855 22167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:42.119678 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:42.119694 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:42.119922 22167 server_base.cc:1061] running on GCE node
W20260501 02:02:42.120083 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:42.120585 22167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:42.121213 22167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:42.122399 22167 hybrid_clock.cc:648] HybridClock initialized: now 1777600962122388 us; error 31 us; skew 500 ppm
May 01 02:02:42 dist-test-slave-2j90 krb5kdc[21783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:42.124965 22167 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:42.125914 22167 webserver.cc:492] Webserver started at http://127.18.96.195:40611/ using document root <none> and password file <none>
I20260501 02:02:42.126506 22167 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:42.126577 22167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:42.126763 22167 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:42.128376 22167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "3ab2d85c61f4432eb179849868996ddf"
format_stamp: "Formatted at 2026-05-01 02:02:42 on dist-test-slave-2j90"
server_key: "d027ab26844ea2f8e3c0b04b1fc778d4"
server_key_iv: "3eb2f5fabb0e54ca268606ec0fe14d0b"
server_key_version: "encryptionkey@0"
I20260501 02:02:42.128842 22167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "3ab2d85c61f4432eb179849868996ddf"
format_stamp: "Formatted at 2026-05-01 02:02:42 on dist-test-slave-2j90"
server_key: "d027ab26844ea2f8e3c0b04b1fc778d4"
server_key_iv: "3eb2f5fabb0e54ca268606ec0fe14d0b"
server_key_version: "encryptionkey@0"
I20260501 02:02:42.131819 22167 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260501 02:02:42.133842 22182 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:42.134894 22167 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260501 02:02:42.135023 22167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "3ab2d85c61f4432eb179849868996ddf"
format_stamp: "Formatted at 2026-05-01 02:02:42 on dist-test-slave-2j90"
server_key: "d027ab26844ea2f8e3c0b04b1fc778d4"
server_key_iv: "3eb2f5fabb0e54ca268606ec0fe14d0b"
server_key_version: "encryptionkey@0"
I20260501 02:02:42.135118 22167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:42.146158 22167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:42.148882 22167 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:42.149089 22167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:42.149621 22167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:42.150498 22167 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:42.150573 22167 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:42.150653 22167 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:42.150688 22167 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:42.159287 22167 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:45763
I20260501 02:02:42.159309 22295 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:45763 every 8 connection(s)
I20260501 02:02:42.160239 22167 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:42.162485 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22167
I20260501 02:02:42.162617 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:42.162909 18819 external_mini_cluster.cc:1468] Setting key fa0d810cae6488d2c9ea9a6135ed52fe
May 01 02:02:42 dist-test-slave-2j90 krb5kdc[21783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600962, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:42.171850 21870 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:42.161800 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:38909 (local address 127.18.96.254:46209)
0501 02:02:42.161981 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:42.161986 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:42.162786 (+ 800us) server_negotiation.cc:408] Connection header received
0501 02:02:42.163627 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:42.163632 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:42.163691 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:42.163794 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:42.165144 (+ 1350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:42.165676 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:42.166307 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:42.166474 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:42.168904 (+ 2430us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:42.168918 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:42.168920 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:42.168943 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:42.170440 (+ 1497us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:42.170831 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:42.170834 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:42.170835 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:42.170872 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:42.171098 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:42.171100 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:42.171102 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:42.171217 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:42.171284 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:42.171661 (+ 377us) server_negotiation.cc:300] Negotiation successful
0501 02:02:42.171740 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:02:42.172430 22298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:42.162135 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46209 (local address 127.18.96.195:38909)
0501 02:02:42.162631 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:42.162672 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:42.163461 (+ 789us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:42.163963 (+ 502us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:42.163973 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:42.164467 (+ 494us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:42.164978 (+ 511us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:42.164990 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:42.165773 (+ 783us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:42.165776 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:42.166195 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:42.166202 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:42.166339 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:42.167394 (+ 1055us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:42.167412 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:42.168776 (+ 1364us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:42.170533 (+ 1757us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:42.170537 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:42.170545 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:42.170749 (+ 204us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:42.170958 (+ 209us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:42.170960 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:42.170961 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:42.171031 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:42.171293 (+ 262us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:42.171297 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:42.171474 (+ 177us) client_negotiation.cc:770] Sending connection context
0501 02:02:42.171621 (+ 147us) client_negotiation.cc:241] Negotiation successful
0501 02:02:42.171771 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":181,"threads_started":1}
I20260501 02:02:42.173368 22296 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46209
I20260501 02:02:42.173595 22296 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:42.174140 22296 heartbeater.cc:507] Master 127.18.96.254:46209 requested a full tablet report, sending...
I20260501 02:02:42.175061 21831 ts_manager.cc:194] Registered new tserver with Master: 3ab2d85c61f4432eb179849868996ddf (127.18.96.195:45763)
I20260501 02:02:42.175765 21831 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:38909
I20260501 02:02:42.176602 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260501 02:02:42.296689 21870 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:42.293436 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42680 (local address 127.18.96.254:46209)
0501 02:02:42.293609 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:42.293614 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:42.294443 (+ 829us) server_negotiation.cc:408] Connection header received
0501 02:02:42.295249 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:42.295252 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:42.295300 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:42.295404 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:42.296564 (+ 1160us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42680: BlockingRecv error: recv got EOF from 127.0.0.1:42680 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260501 02:02:42.297812 21870 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:42.296996 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.18.96.254:46209)
0501 02:02:42.297126 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:42.297130 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:42.297202 (+ 72us) server_negotiation.cc:408] Connection header received
0501 02:02:42.297285 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:42.297287 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:42.297320 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:42.297383 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:42.297726 (+ 343us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42686: BlockingRecv error: recv got EOF from 127.0.0.1:42686 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260501 02:02:42.298801 21870 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:42.298077 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.18.96.254:46209)
0501 02:02:42.298163 (+ 86us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:42.298168 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:42.298248 (+ 80us) server_negotiation.cc:408] Connection header received
0501 02:02:42.298336 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:42.298338 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:42.298367 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:42.298437 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:42.298725 (+ 288us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42692: BlockingRecv error: recv got EOF from 127.0.0.1:42692 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260501 02:02:42.302084 21870 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:42.301372 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.18.96.254:46209)
0501 02:02:42.301486 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:42.301490 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:42.301503 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:02:42.301560 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:42.301563 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:42.301596 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:42.301679 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:42.302017 (+ 338us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42694: BlockingRecv error: recv got EOF from 127.0.0.1:42694 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:02:42.309827 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21895
I20260501 02:02:42.315515 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22031
I20260501 02:02:42.320721 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22167
I20260501 02:02:42.326172 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 21799
2026-05-01T02:02:42Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3123 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:42 dist-test-slave-2j90 krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:42 dist-test-slave-2j90 krb5kdc[22319](info): set up 2 sockets
May 01 02:02:42 dist-test-slave-2j90 krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260501 02:02:44.298521 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:44.426685 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46365
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:44.522912 22335 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:44.523147 22335 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:44.523227 22335 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:44.526496 22335 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:44.526589 22335 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:44.526634 22335 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:44.526695 22335 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:44.526741 22335 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:44.530740 22335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:46365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:44.531754 22335 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:44.532508 22335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:44.537744 22341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:44.537792 22335 server_base.cc:1061] running on GCE node
W20260501 02:02:44.537740 22340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:44.538021 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:44.538408 22335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:44.539219 22335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:44.540382 22335 hybrid_clock.cc:648] HybridClock initialized: now 1777600964540332 us; error 61 us; skew 500 ppm
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600964, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:44.542915 22335 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:44.543893 22335 webserver.cc:492] Webserver started at http://127.18.96.254:41531/ using document root <none> and password file <none>
I20260501 02:02:44.544392 22335 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:44.544484 22335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:44.544693 22335 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:44.546288 22335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "ac5141240e6c41f1991723407057bd7d"
format_stamp: "Formatted at 2026-05-01 02:02:44 on dist-test-slave-2j90"
server_key: "818ffebc75d49102bc7696fface4e20b"
server_key_iv: "0e76d39c50bf142ed801c91cca5a869a"
server_key_version: "encryptionkey@0"
I20260501 02:02:44.546758 22335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "ac5141240e6c41f1991723407057bd7d"
format_stamp: "Formatted at 2026-05-01 02:02:44 on dist-test-slave-2j90"
server_key: "818ffebc75d49102bc7696fface4e20b"
server_key_iv: "0e76d39c50bf142ed801c91cca5a869a"
server_key_version: "encryptionkey@0"
I20260501 02:02:44.549645 22335 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:02:44.551532 22350 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:44.552399 22335 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:44.552542 22335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "ac5141240e6c41f1991723407057bd7d"
format_stamp: "Formatted at 2026-05-01 02:02:44 on dist-test-slave-2j90"
server_key: "818ffebc75d49102bc7696fface4e20b"
server_key_iv: "0e76d39c50bf142ed801c91cca5a869a"
server_key_version: "encryptionkey@0"
I20260501 02:02:44.552628 22335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:44.568522 22335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:44.571149 22335 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:44.571385 22335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:44.578353 22402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:46365 every 8 connection(s)
I20260501 02:02:44.578353 22335 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:46365
I20260501 02:02:44.579347 22335 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:44.581504 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22335
I20260501 02:02:44.581633 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:44.581852 18819 external_mini_cluster.cc:1468] Setting key aba5d4965ffebb28965cbcd586cec821
I20260501 02:02:44.581869 22403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:44.587913 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: Bootstrap starting.
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:44.589833 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:44.590559 22403 log.cc:826] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:44.592042 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: No bootstrap required, opened a new log
I20260501 02:02:44.593727 22406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:44.582966 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.18.96.254:46365)
0501 02:02:44.583361 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:44.583369 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:44.583399 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:02:44.584570 (+ 1171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:44.584587 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:44.584838 (+ 251us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:44.585064 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:44.586124 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:44.586830 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:44.587592 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:44.587793 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:44.589857 (+ 2064us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:44.589873 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:44.589882 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:44.589906 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:44.591567 (+ 1661us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:44.591983 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:44.591987 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:44.591991 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:44.592049 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:44.592315 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:44.592317 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:44.592318 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:44.592573 (+ 255us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:44.592704 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:44.592974 (+ 270us) server_negotiation.cc:300] Negotiation successful
0501 02:02:44.593163 (+ 189us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260501 02:02:44.594520 22403 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } }
I20260501 02:02:44.594720 22403 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:44.594802 22403 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5141240e6c41f1991723407057bd7d, State: Initialized, Role: FOLLOWER
I20260501 02:02:44.595170 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } }
I20260501 02:02:44.595306 22403 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:44.595361 22403 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:44.595458 22403 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:44.596299 22403 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } }
I20260501 02:02:44.596556 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:02:44.596661 22403 leader_election.cc:304] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5141240e6c41f1991723407057bd7d; no voters:
I20260501 02:02:44.596925 22403 leader_election.cc:290] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:44.597039 22408 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:44.597256 22408 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [term 1 LEADER]: Becoming Leader. State: Replica: ac5141240e6c41f1991723407057bd7d, State: Running, Role: LEADER
I20260501 02:02:44.597534 22408 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } }
I20260501 02:02:44.597904 22403 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:44.598614 22408 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5141240e6c41f1991723407057bd7d. Latest consensus state: current_term: 1 leader_uuid: "ac5141240e6c41f1991723407057bd7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } } }
I20260501 02:02:44.598737 22408 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:44.598932 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5141240e6c41f1991723407057bd7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5141240e6c41f1991723407057bd7d" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46365 } } }
I20260501 02:02:44.599020 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d [sys.catalog]: This master's current role is: LEADER
W20260501 02:02:44.602725 22424 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:02:44.602809 22424 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:02:44.602903 22415 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:44.603629 22415 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:44.609287 22415 catalog_manager.cc:1357] Generated new cluster ID: a4c99bc71eab48ea80c27085d0119988
I20260501 02:02:44.609397 22415 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:44.614408 22406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:44.609918 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.18.96.254:46365)
0501 02:02:44.610105 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:44.610110 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:44.610365 (+ 255us) server_negotiation.cc:408] Connection header received
0501 02:02:44.610487 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:44.610490 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:44.610536 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:44.610624 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:44.611380 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:44.612062 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:44.612890 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:44.613367 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:44.613546 (+ 179us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:44.613559 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:44.613562 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:44.614011 (+ 449us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:44.614039 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:44.614053 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:44.614130 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:44.614250 (+ 120us) server_negotiation.cc:300] Negotiation successful
0501 02:02:44.614292 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260501 02:02:44.624583 22415 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:44.625320 22415 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:44.637399 22415 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ac5141240e6c41f1991723407057bd7d: Generated new TSK 0
I20260501 02:02:44.637943 22415 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:02:44.644908 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22335
2026-05-01T02:02:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2316 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22436](info): set up 2 sockets
May 01 02:02:44 dist-test-slave-2j90 krb5kdc[22436](info): commencing operation
krb5kdc: starting...
W20260501 02:02:46.630527 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:46 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:46.762737 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:42349
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:42349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:46.859089 22452 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:46.859314 22452 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:46.859401 22452 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:46.862778 22452 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:46.862838 22452 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:46.862861 22452 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:46.862905 22452 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:46.862939 22452 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:46.867236 22452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:42349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:42349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:46.868309 22452 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:46.869190 22452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:46.874904 22457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:46.874904 22460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:46.874998 22458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:46.875221 22452 server_base.cc:1061] running on GCE node
I20260501 02:02:46.875875 22452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:46.876771 22452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:46.877995 22452 hybrid_clock.cc:648] HybridClock initialized: now 1777600966877931 us; error 44 us; skew 500 ppm
May 01 02:02:46 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600966, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:46.880609 22452 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:46.881664 22452 webserver.cc:492] Webserver started at http://127.18.96.254:39991/ using document root <none> and password file <none>
I20260501 02:02:46.882267 22452 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:46.882349 22452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:46.882539 22452 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:46.884145 22452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "f2e11b007fb448b6ad8ef87f47511472"
format_stamp: "Formatted at 2026-05-01 02:02:46 on dist-test-slave-2j90"
server_key: "2b88af88c68f69897e2a693d99b42e4d"
server_key_iv: "11467b6d16fc024396995fd6459c634b"
server_key_version: "encryptionkey@0"
I20260501 02:02:46.884629 22452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "f2e11b007fb448b6ad8ef87f47511472"
format_stamp: "Formatted at 2026-05-01 02:02:46 on dist-test-slave-2j90"
server_key: "2b88af88c68f69897e2a693d99b42e4d"
server_key_iv: "11467b6d16fc024396995fd6459c634b"
server_key_version: "encryptionkey@0"
I20260501 02:02:46.887815 22452 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260501 02:02:46.889817 22467 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:46.890761 22452 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:46.890890 22452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "f2e11b007fb448b6ad8ef87f47511472"
format_stamp: "Formatted at 2026-05-01 02:02:46 on dist-test-slave-2j90"
server_key: "2b88af88c68f69897e2a693d99b42e4d"
server_key_iv: "11467b6d16fc024396995fd6459c634b"
server_key_version: "encryptionkey@0"
I20260501 02:02:46.890990 22452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:46.909873 22452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:46.915896 22452 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:46.916118 22452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:46.923533 22452 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:42349
I20260501 02:02:46.923552 22519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:42349 every 8 connection(s)
I20260501 02:02:46.924544 22452 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:46.927258 22520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:46.927812 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22452
I20260501 02:02:46.928045 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:46.928287 18819 external_mini_cluster.cc:1468] Setting key 01a285a2eca543a354004317b39e0467
I20260501 02:02:46.933779 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472: Bootstrap starting.
May 01 02:02:46 dist-test-slave-2j90 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:46.935873 22520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:46.936728 22520 log.cc:826] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:46.938408 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472: No bootstrap required, opened a new log
I20260501 02:02:46.940392 22523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:46.929540 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.18.96.254:42349)
0501 02:02:46.929983 (+ 443us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:46.929999 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:46.930048 (+ 49us) server_negotiation.cc:408] Connection header received
0501 02:02:46.930705 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:46.930729 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:46.931086 (+ 357us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:46.931414 (+ 328us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:46.932174 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:46.933299 (+ 1125us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:46.933945 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:46.934209 (+ 264us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:46.936301 (+ 2092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:46.936319 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:46.936332 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:46.936362 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:46.938100 (+ 1738us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:46.938505 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:46.938511 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:46.938517 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:46.938596 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:46.938858 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:46.938873 (+ 15us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:46.938875 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:46.939154 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:46.939340 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:46.939631 (+ 291us) server_negotiation.cc:300] Negotiation successful
0501 02:02:46.939844 (+ 213us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":101,"threads_started":1}
I20260501 02:02:46.940924 22520 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } }
I20260501 02:02:46.941118 22520 raft_consensus.cc:385] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:46.941198 22520 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e11b007fb448b6ad8ef87f47511472, State: Initialized, Role: FOLLOWER
I20260501 02:02:46.941635 22520 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } }
I20260501 02:02:46.941745 22520 raft_consensus.cc:399] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:46.941824 22520 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:46.941947 22520 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:46.942948 22520 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } }
I20260501 02:02:46.943279 22520 leader_election.cc:304] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2e11b007fb448b6ad8ef87f47511472; no voters:
I20260501 02:02:46.943545 22520 leader_election.cc:290] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:46.943684 22525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:46.943889 22525 raft_consensus.cc:697] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [term 1 LEADER]: Becoming Leader. State: Replica: f2e11b007fb448b6ad8ef87f47511472, State: Running, Role: LEADER
I20260501 02:02:46.944150 22525 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } }
I20260501 02:02:46.944644 22520 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:46.945166 22525 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e11b007fb448b6ad8ef87f47511472. Latest consensus state: current_term: 1 leader_uuid: "f2e11b007fb448b6ad8ef87f47511472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } } }
I20260501 02:02:46.945317 22525 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:46.945578 22527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2e11b007fb448b6ad8ef87f47511472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2e11b007fb448b6ad8ef87f47511472" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 42349 } } }
I20260501 02:02:46.945717 22529 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:46.945739 22527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:46.948588 22529 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:46.953651 22529 catalog_manager.cc:1357] Generated new cluster ID: 387d5fc66ccf44fb8b2d09249ab2a7c3
I20260501 02:02:46.953724 22529 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:46.959704 22529 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:46.960461 22529 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:46.966130 22529 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f2e11b007fb448b6ad8ef87f47511472: Generated new TSK 0
I20260501 02:02:46.966632 22529 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:47.019276 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:42349
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:47.118602 22548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:47.118839 22548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:47.118921 22548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:47.122210 22548 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:47.122301 22548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:47.122421 22548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:47.126376 22548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:42349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:47.127548 22548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:47.128592 22548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:47.135885 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:47.135922 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:47.135885 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:47.137030 22548 server_base.cc:1061] running on GCE node
I20260501 02:02:47.137395 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:47.137908 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:47.139122 22548 hybrid_clock.cc:648] HybridClock initialized: now 1777600967139093 us; error 41 us; skew 500 ppm
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:47.141860 22548 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:47.142962 22548 webserver.cc:492] Webserver started at http://127.18.96.193:45317/ using document root <none> and password file <none>
I20260501 02:02:47.143512 22548 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:47.143592 22548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:47.143780 22548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:47.145362 22548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "583c3749dd0949508555061b0282799f"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "e1e67055cf031259a8ac604d8f704bff"
server_key_iv: "179dd62b254a073c774851548283fedf"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.145834 22548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "583c3749dd0949508555061b0282799f"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "e1e67055cf031259a8ac604d8f704bff"
server_key_iv: "179dd62b254a073c774851548283fedf"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.149201 22548 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:47.151330 22563 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.152241 22548 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:47.152365 22548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "583c3749dd0949508555061b0282799f"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "e1e67055cf031259a8ac604d8f704bff"
server_key_iv: "179dd62b254a073c774851548283fedf"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.152463 22548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:47.168586 22548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:47.171098 22548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:47.171293 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:47.171833 22548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:47.172690 22548 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:47.172760 22548 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.172838 22548 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:47.172890 22548 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.182122 22548 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:36623
I20260501 02:02:47.182179 22676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:36623 every 8 connection(s)
I20260501 02:02:47.183060 22548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:47.184867 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22548
I20260501 02:02:47.185011 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:47.185251 18819 external_mini_cluster.cc:1468] Setting key cbcc5a7fe529387382864a67a55a61d5
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:47.196655 22523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.184863 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:36095 (local address 127.18.96.254:42349)
0501 02:02:47.185046 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:47.185051 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:47.185791 (+ 740us) server_negotiation.cc:408] Connection header received
0501 02:02:47.186753 (+ 962us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:47.186756 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:47.186798 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:47.186866 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:47.188355 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.189211 (+ 856us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.189848 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.190093 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.193120 (+ 3027us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:47.193141 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:47.193146 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:47.193180 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:47.194884 (+ 1704us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.195422 (+ 538us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.195425 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.195426 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.195469 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.195727 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.195729 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.195730 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.195856 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:47.195929 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:47.196414 (+ 485us) server_negotiation.cc:300] Negotiation successful
0501 02:02:47.196510 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260501 02:02:47.197263 22679 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.185131 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:42349 (local address 127.18.96.193:36095)
0501 02:02:47.185628 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:47.185686 (+ 58us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:47.186555 (+ 869us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:47.187039 (+ 484us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:47.187050 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:47.187602 (+ 552us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:47.188196 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.188213 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.189331 (+ 1118us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.189334 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:47.189715 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.189720 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.189876 (+ 156us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.191196 (+ 1320us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:47.191223 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:47.192971 (+ 1748us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:47.195014 (+ 2043us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.195020 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.195032 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.195312 (+ 280us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.195557 (+ 245us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.195559 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.195561 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.195651 (+ 90us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.195940 (+ 289us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:47.195943 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:47.196215 (+ 272us) client_negotiation.cc:770] Sending connection context
0501 02:02:47.196372 (+ 157us) client_negotiation.cc:241] Negotiation successful
0501 02:02:47.196559 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:02:47.198539 22677 heartbeater.cc:344] Connected to a master server at 127.18.96.254:42349
I20260501 02:02:47.198791 22677 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:47.199355 22677 heartbeater.cc:507] Master 127.18.96.254:42349 requested a full tablet report, sending...
I20260501 02:02:47.200934 22484 ts_manager.cc:194] Registered new tserver with Master: 583c3749dd0949508555061b0282799f (127.18.96.193:36623)
I20260501 02:02:47.202658 22484 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:36095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:47.234416 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:42349
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:47.332350 22684 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:47.332532 22684 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:47.332583 22684 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:47.335769 22684 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:47.335822 22684 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:47.335898 22684 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:47.339712 22684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:42349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:47.340588 22684 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:47.341306 22684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:47.347304 22689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:47.347378 22690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:47.347416 22684 server_base.cc:1061] running on GCE node
W20260501 02:02:47.347296 22692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:47.347864 22684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:47.348376 22684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:47.349551 22684 hybrid_clock.cc:648] HybridClock initialized: now 1777600967349530 us; error 33 us; skew 500 ppm
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:47.352201 22684 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:47.353178 22684 webserver.cc:492] Webserver started at http://127.18.96.194:37947/ using document root <none> and password file <none>
I20260501 02:02:47.353710 22684 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:47.353783 22684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:47.354066 22684 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:47.355659 22684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "c1b8ee021ccd4cf68397f74d6ee25ca6"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "14088d7d77ddd8e5fa55d8f132564be2"
server_key_iv: "205fa3dbb7bfaa3f43e8bfcf01e9f78f"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.356120 22684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "c1b8ee021ccd4cf68397f74d6ee25ca6"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "14088d7d77ddd8e5fa55d8f132564be2"
server_key_iv: "205fa3dbb7bfaa3f43e8bfcf01e9f78f"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.359385 22684 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:47.361336 22699 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.362264 22684 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:47.362404 22684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "c1b8ee021ccd4cf68397f74d6ee25ca6"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "14088d7d77ddd8e5fa55d8f132564be2"
server_key_iv: "205fa3dbb7bfaa3f43e8bfcf01e9f78f"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.362495 22684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:47.372623 22684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:47.375026 22684 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:47.375217 22684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:47.375763 22684 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:47.376612 22684 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:47.376683 22684 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.376753 22684 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:47.376802 22684 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.385843 22684 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:44775
I20260501 02:02:47.385865 22812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:44775 every 8 connection(s)
I20260501 02:02:47.386803 22684 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:47.389158 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22684
I20260501 02:02:47.389282 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:47.389559 18819 external_mini_cluster.cc:1468] Setting key 3e22a7575df7f2cfd07ff2db187c61c8
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:47.399842 22523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.388401 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:35285 (local address 127.18.96.254:42349)
0501 02:02:47.388589 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:47.388595 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:47.389303 (+ 708us) server_negotiation.cc:408] Connection header received
0501 02:02:47.390456 (+ 1153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:47.390460 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:47.390505 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:47.390600 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:47.392110 (+ 1510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.392644 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.393316 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.393489 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.396463 (+ 2974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:47.396485 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:47.396487 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:47.396511 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:47.397930 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.398438 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.398442 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.398444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.398491 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.398820 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.398823 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.398825 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.398964 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:47.399063 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:47.399525 (+ 462us) server_negotiation.cc:300] Negotiation successful
0501 02:02:47.399675 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:02:47.400379 22815 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.388660 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:42349 (local address 127.18.96.194:35285)
0501 02:02:47.389109 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:47.389152 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:47.390241 (+ 1089us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:47.390736 (+ 495us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:47.390744 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:47.391308 (+ 564us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:47.391960 (+ 652us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.391979 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.392763 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.392768 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:47.393192 (+ 424us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.393198 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.393390 (+ 192us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.394547 (+ 1157us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:47.394565 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:47.396318 (+ 1753us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:47.398091 (+ 1773us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.398096 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.398108 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.398327 (+ 219us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.398597 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.398600 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.398601 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.398725 (+ 124us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.399071 (+ 346us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:47.399079 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:47.399276 (+ 197us) client_negotiation.cc:770] Sending connection context
0501 02:02:47.399480 (+ 204us) client_negotiation.cc:241] Negotiation successful
0501 02:02:47.399703 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:47.401473 22813 heartbeater.cc:344] Connected to a master server at 127.18.96.254:42349
I20260501 02:02:47.401757 22813 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:47.402382 22813 heartbeater.cc:507] Master 127.18.96.254:42349 requested a full tablet report, sending...
I20260501 02:02:47.403475 22484 ts_manager.cc:194] Registered new tserver with Master: c1b8ee021ccd4cf68397f74d6ee25ca6 (127.18.96.194:44775)
I20260501 02:02:47.404033 22484 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:35285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:47.438688 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:42349
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:47.537990 22820 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:47.538184 22820 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:47.538292 22820 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:47.541481 22820 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:47.541540 22820 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:47.541684 22820 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:47.545818 22820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:42349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:47.547039 22820 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:47.547854 22820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:47.554085 22826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:47.554092 22828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:47.554086 22825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:47.554160 22820 server_base.cc:1061] running on GCE node
I20260501 02:02:47.554648 22820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:47.555154 22820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:47.556357 22820 hybrid_clock.cc:648] HybridClock initialized: now 1777600967556336 us; error 42 us; skew 500 ppm
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:47.559019 22820 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:47.559954 22820 webserver.cc:492] Webserver started at http://127.18.96.195:38383/ using document root <none> and password file <none>
I20260501 02:02:47.560474 22820 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:47.560544 22820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:47.560715 22820 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:47.562378 22820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "d8f8885f28564ff883d77ae051dce91a"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "fe35d80efd8ab57340c46e8846ea5b4d"
server_key_iv: "0a65082c5f1b18332c6914e3d795bc51"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.562841 22820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f8885f28564ff883d77ae051dce91a"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "fe35d80efd8ab57340c46e8846ea5b4d"
server_key_iv: "0a65082c5f1b18332c6914e3d795bc51"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.565882 22820 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:02:47.567916 22835 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.568792 22820 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:47.568914 22820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "d8f8885f28564ff883d77ae051dce91a"
format_stamp: "Formatted at 2026-05-01 02:02:47 on dist-test-slave-2j90"
server_key: "fe35d80efd8ab57340c46e8846ea5b4d"
server_key_iv: "0a65082c5f1b18332c6914e3d795bc51"
server_key_version: "encryptionkey@0"
I20260501 02:02:47.569006 22820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:47.579622 22820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:47.581943 22820 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:47.582185 22820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:47.582708 22820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:47.583536 22820 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:47.583609 22820 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.583674 22820 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:47.583729 22820 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:47.592353 22820 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:38853
I20260501 02:02:47.592407 22948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:38853 every 8 connection(s)
I20260501 02:02:47.593232 22820 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:47.593847 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22820
I20260501 02:02:47.594017 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:47.594280 18819 external_mini_cluster.cc:1468] Setting key d41ff224d7a09f596aee44a26cc07167
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:47.604985 22523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.595108 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:58431 (local address 127.18.96.254:42349)
0501 02:02:47.595232 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:47.595236 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:47.595802 (+ 566us) server_negotiation.cc:408] Connection header received
0501 02:02:47.596699 (+ 897us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:47.596702 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:47.596755 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:47.596840 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:47.597877 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.598516 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.599118 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.599250 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.601861 (+ 2611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:47.601877 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:47.601879 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:47.601907 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:47.603329 (+ 1422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.603775 (+ 446us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.603778 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.603780 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.603826 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.604111 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.604115 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.604116 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.604252 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:47.604342 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:47.604758 (+ 416us) server_negotiation.cc:300] Negotiation successful
0501 02:02:47.604857 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260501 02:02:47.605471 22951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.595242 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:42349 (local address 127.18.96.195:58431)
0501 02:02:47.595669 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:47.595708 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:47.596470 (+ 762us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:47.596976 (+ 506us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:47.596983 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:47.597321 (+ 338us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:47.597725 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.597735 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.598627 (+ 892us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.598629 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:47.599018 (+ 389us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:47.599024 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.599163 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.600183 (+ 1020us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:47.600200 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:47.601700 (+ 1500us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:47.603443 (+ 1743us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.603448 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.603456 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.603671 (+ 215us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.603913 (+ 242us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:47.603916 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:47.603917 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:47.603993 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:47.604347 (+ 354us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:47.604351 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:47.604523 (+ 172us) client_negotiation.cc:770] Sending connection context
0501 02:02:47.604672 (+ 149us) client_negotiation.cc:241] Negotiation successful
0501 02:02:47.604838 (+ 166us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260501 02:02:47.606647 22949 heartbeater.cc:344] Connected to a master server at 127.18.96.254:42349
I20260501 02:02:47.606931 22949 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:47.607509 22949 heartbeater.cc:507] Master 127.18.96.254:42349 requested a full tablet report, sending...
I20260501 02:02:47.608637 22484 ts_manager.cc:194] Registered new tserver with Master: d8f8885f28564ff883d77ae051dce91a (127.18.96.195:38853)
I20260501 02:02:47.609174 22484 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:58431
I20260501 02:02:47.620046 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600967, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:47.642587 22523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:47.635358 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36158 (local address 127.18.96.254:42349)
0501 02:02:47.635569 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:47.635573 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:47.635588 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:02:47.635713 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:47.635716 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:47.635762 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:47.635846 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:47.636575 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.637049 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:47.637693 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:47.637834 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:47.639826 (+ 1992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:47.639843 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:47.639845 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:47.639868 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:47.641264 (+ 1396us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.641578 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.641581 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.641582 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.641619 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:47.641802 (+ 183us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:47.641804 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:47.641805 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:47.641948 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:47.642106 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:47.642265 (+ 159us) server_negotiation.cc:300] Negotiation successful
0501 02:02:47.642365 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260501 02:02:47.642961 22484 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36158
I20260501 02:02:47.643752 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22548
I20260501 02:02:47.649515 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22684
I20260501 02:02:47.655053 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22820
I20260501 02:02:47.660414 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22452
2026-05-01T02:02:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3017 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22961](info): set up 2 sockets
May 01 02:02:47 dist-test-slave-2j90 krb5kdc[22961](info): commencing operation
krb5kdc: starting...
W20260501 02:02:49.662998 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.981s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:49 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:49.793161 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44833
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:49.889842 22977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:49.890084 22977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:49.890165 22977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:49.893361 22977 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:49.893450 22977 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:49.893503 22977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:49.893563 22977 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:49.893599 22977 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:49.897658 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:49.898638 22977 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:49.899412 22977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:49.904739 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:49.904743 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:49.904747 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:49.904940 22977 server_base.cc:1061] running on GCE node
I20260501 02:02:49.905489 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:49.906417 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:49.907619 22977 hybrid_clock.cc:648] HybridClock initialized: now 1777600969907594 us; error 36 us; skew 500 ppm
May 01 02:02:49 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600969, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:49.910225 22977 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:49.911234 22977 webserver.cc:492] Webserver started at http://127.18.96.254:43731/ using document root <none> and password file <none>
I20260501 02:02:49.911798 22977 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:49.911870 22977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:49.912076 22977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:49.913646 22977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "5c70690b61e44a329e73bbaee5fb7684"
format_stamp: "Formatted at 2026-05-01 02:02:49 on dist-test-slave-2j90"
server_key: "446fcb61df918b4e3fe32dafef272e21"
server_key_iv: "37ae0f0d056c9df52ecaa14fc56488d8"
server_key_version: "encryptionkey@0"
I20260501 02:02:49.914165 22977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "5c70690b61e44a329e73bbaee5fb7684"
format_stamp: "Formatted at 2026-05-01 02:02:49 on dist-test-slave-2j90"
server_key: "446fcb61df918b4e3fe32dafef272e21"
server_key_iv: "37ae0f0d056c9df52ecaa14fc56488d8"
server_key_version: "encryptionkey@0"
I20260501 02:02:49.917245 22977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:02:49.919263 22992 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:49.920189 22977 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:49.920311 22977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "5c70690b61e44a329e73bbaee5fb7684"
format_stamp: "Formatted at 2026-05-01 02:02:49 on dist-test-slave-2j90"
server_key: "446fcb61df918b4e3fe32dafef272e21"
server_key_iv: "37ae0f0d056c9df52ecaa14fc56488d8"
server_key_version: "encryptionkey@0"
I20260501 02:02:49.920403 22977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:49.935451 22977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:49.937915 22977 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:49.938131 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:49.945509 22977 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44833
I20260501 02:02:49.945509 23044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44833 every 8 connection(s)
I20260501 02:02:49.946524 22977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:49.947640 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 22977
I20260501 02:02:49.947764 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:49.948046 18819 external_mini_cluster.cc:1468] Setting key 6e45e14bf5bba16415c90785c50d040b
I20260501 02:02:49.949559 23045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:49.954078 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684: Bootstrap starting.
May 01 02:02:49 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:49.956276 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:49.957038 23045 log.cc:826] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:49.958871 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684: No bootstrap required, opened a new log
I20260501 02:02:49.960225 23048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:49.949270 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.18.96.254:44833)
0501 02:02:49.949688 (+ 418us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:49.949697 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:49.949727 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:02:49.950851 (+ 1124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:49.950869 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:49.951124 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:49.951422 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:49.952298 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:49.953012 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:49.953650 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:49.953934 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:49.956215 (+ 2281us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:49.956230 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:49.956240 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:49.956263 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:49.958075 (+ 1812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:49.958495 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:49.958499 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:49.958502 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:49.958558 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:49.958782 (+ 224us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:49.958784 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:49.958785 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:49.959089 (+ 304us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:49.959209 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:49.959477 (+ 268us) server_negotiation.cc:300] Negotiation successful
0501 02:02:49.959673 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260501 02:02:49.961427 23045 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } }
I20260501 02:02:49.961625 23045 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:49.961671 23045 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c70690b61e44a329e73bbaee5fb7684, State: Initialized, Role: FOLLOWER
I20260501 02:02:49.962100 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } }
I20260501 02:02:49.962236 23045 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:49.962318 23045 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:49.962419 23045 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:49.963620 23045 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } }
I20260501 02:02:49.963948 23045 leader_election.cc:304] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c70690b61e44a329e73bbaee5fb7684; no voters:
I20260501 02:02:49.964210 23045 leader_election.cc:290] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:49.964339 23050 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:49.964540 23050 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [term 1 LEADER]: Becoming Leader. State: Replica: 5c70690b61e44a329e73bbaee5fb7684, State: Running, Role: LEADER
I20260501 02:02:49.964895 23050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } }
I20260501 02:02:49.965266 23045 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:49.965932 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c70690b61e44a329e73bbaee5fb7684. Latest consensus state: current_term: 1 leader_uuid: "5c70690b61e44a329e73bbaee5fb7684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } } }
I20260501 02:02:49.966145 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:49.966323 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c70690b61e44a329e73bbaee5fb7684" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c70690b61e44a329e73bbaee5fb7684" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44833 } } }
I20260501 02:02:49.966409 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:49.966614 23059 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:49.969451 23059 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:49.973927 23059 catalog_manager.cc:1357] Generated new cluster ID: 346ee327b69446ed91e3e759ef2cdf52
I20260501 02:02:49.974026 23059 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:49.981613 23059 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:49.982422 23059 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:49.988365 23059 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5c70690b61e44a329e73bbaee5fb7684: Generated new TSK 0
I20260501 02:02:49.988852 23059 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:50.043095 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44833
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:50.140379 23073 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:50.140568 23073 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:50.140668 23073 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:50.143918 23073 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:50.143970 23073 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:50.144098 23073 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:50.147977 23073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:50.148864 23073 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:50.149613 23073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 02:02:50.155612 23073 server_base.cc:1061] running on GCE node
W20260501 02:02:50.155542 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:50.155620 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:50.155541 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:50.156064 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:50.156574 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:50.157712 23073 hybrid_clock.cc:648] HybridClock initialized: now 1777600970157690 us; error 38 us; skew 500 ppm
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:50.160274 23073 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:50.161181 23073 webserver.cc:492] Webserver started at http://127.18.96.193:40201/ using document root <none> and password file <none>
I20260501 02:02:50.161680 23073 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:50.161727 23073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:50.161931 23073 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:50.163483 23073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "e48407fa3de742e0b5bb20e8915c679f"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "8ab0cafe6042a814b9d5faf634060ada"
server_key_iv: "de830a9ebe6ff805efdd88330eb4d57e"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.163954 23073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "e48407fa3de742e0b5bb20e8915c679f"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "8ab0cafe6042a814b9d5faf634060ada"
server_key_iv: "de830a9ebe6ff805efdd88330eb4d57e"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.167052 23073 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260501 02:02:50.169068 23088 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.170004 23073 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260501 02:02:50.170112 23073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "e48407fa3de742e0b5bb20e8915c679f"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "8ab0cafe6042a814b9d5faf634060ada"
server_key_iv: "de830a9ebe6ff805efdd88330eb4d57e"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.170210 23073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:50.198386 23073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:50.200662 23073 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:50.200860 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:50.201375 23073 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:50.202282 23073 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:50.202329 23073 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.202392 23073 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:50.202432 23073 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.211134 23073 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44637
I20260501 02:02:50.211160 23201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44637 every 8 connection(s)
I20260501 02:02:50.211995 23073 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:50.218307 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23073
I20260501 02:02:50.218420 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:50.218684 18819 external_mini_cluster.cc:1468] Setting key a09ae0d44a68823e93ffd0dc1e2c20f0
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:50.224265 23048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.213595 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39879 (local address 127.18.96.254:44833)
0501 02:02:50.213733 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:50.213737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:50.214372 (+ 635us) server_negotiation.cc:408] Connection header received
0501 02:02:50.215194 (+ 822us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:50.215198 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:50.215257 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:50.215355 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:50.216782 (+ 1427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.217270 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.217916 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.218138 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.220454 (+ 2316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:50.220475 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:50.220479 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:50.220515 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:50.222208 (+ 1693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.222824 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.222826 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.222827 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.222868 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.223193 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.223195 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.223196 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.223324 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:50.223398 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:50.223984 (+ 586us) server_negotiation.cc:300] Negotiation successful
0501 02:02:50.224132 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:02:50.225020 23204 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.213828 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44833 (local address 127.18.96.193:39879)
0501 02:02:50.214252 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:50.214280 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:50.215008 (+ 728us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:50.215498 (+ 490us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:50.215505 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:50.215852 (+ 347us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:50.216629 (+ 777us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.216641 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.217392 (+ 751us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.217396 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:50.217782 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.217788 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.218021 (+ 233us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.218616 (+ 595us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:50.218633 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:50.220307 (+ 1674us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:50.222321 (+ 2014us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.222328 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.222342 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.222725 (+ 383us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.222960 (+ 235us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.222967 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.222972 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.223100 (+ 128us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.223430 (+ 330us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:50.223439 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:50.223704 (+ 265us) client_negotiation.cc:770] Sending connection context
0501 02:02:50.223929 (+ 225us) client_negotiation.cc:241] Negotiation successful
0501 02:02:50.224179 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":172,"threads_started":1}
I20260501 02:02:50.226303 23202 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44833
I20260501 02:02:50.226557 23202 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:50.227051 23202 heartbeater.cc:507] Master 127.18.96.254:44833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:50.228441 23009 ts_manager.cc:194] Registered new tserver with Master: e48407fa3de742e0b5bb20e8915c679f (127.18.96.193:44637)
I20260501 02:02:50.229645 23009 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:39879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:50.264756 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44833
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:50.362704 23209 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:50.362926 23209 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:50.363011 23209 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:50.366161 23209 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:50.366250 23209 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:50.366410 23209 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:50.370345 23209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:50.371347 23209 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:50.372134 23209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:50.378136 23217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:50.378186 23214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:50.378363 23209 server_base.cc:1061] running on GCE node
W20260501 02:02:50.378136 23215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:50.378803 23209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:50.379314 23209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:50.380484 23209 hybrid_clock.cc:648] HybridClock initialized: now 1777600970380437 us; error 63 us; skew 500 ppm
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:50.383031 23209 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:50.384001 23209 webserver.cc:492] Webserver started at http://127.18.96.194:46021/ using document root <none> and password file <none>
I20260501 02:02:50.384522 23209 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:50.384624 23209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:50.384828 23209 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:50.386461 23209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "873fff57d063443198dd7f73ca79be75"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "ca91847781f6392cc2b2f270fb0fa3d4"
server_key_iv: "ffb47e1a96f3e94f0a4b5805541cb605"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.386951 23209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "873fff57d063443198dd7f73ca79be75"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "ca91847781f6392cc2b2f270fb0fa3d4"
server_key_iv: "ffb47e1a96f3e94f0a4b5805541cb605"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.390053 23209 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:02:50.392015 23224 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.392956 23209 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:50.393090 23209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "873fff57d063443198dd7f73ca79be75"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "ca91847781f6392cc2b2f270fb0fa3d4"
server_key_iv: "ffb47e1a96f3e94f0a4b5805541cb605"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.393188 23209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:50.408943 23209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:50.411700 23209 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:50.411908 23209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:50.412444 23209 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:50.413306 23209 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:50.413354 23209 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.413391 23209 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:50.413458 23209 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.421618 23209 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:39759
I20260501 02:02:50.421669 23337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:39759 every 8 connection(s)
I20260501 02:02:50.422560 23209 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:50.429752 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23209
I20260501 02:02:50.429908 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:50.430231 18819 external_mini_cluster.cc:1468] Setting key e0bbae5dabdc1306e898d85ad12589fe
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:50.434839 23048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.424168 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40489 (local address 127.18.96.254:44833)
0501 02:02:50.424300 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:50.424304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:50.424947 (+ 643us) server_negotiation.cc:408] Connection header received
0501 02:02:50.425795 (+ 848us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:50.425799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:50.425871 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:50.425984 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:50.427486 (+ 1502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.427980 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.428563 (+ 583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.428743 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.431068 (+ 2325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:50.431095 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:50.431098 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:50.431125 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:50.432883 (+ 1758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.433313 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.433319 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.433322 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.433384 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.433729 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.433732 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.433733 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.433868 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:50.433997 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:50.434629 (+ 632us) server_negotiation.cc:300] Negotiation successful
0501 02:02:50.434723 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:02:50.435464 23340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.424385 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44833 (local address 127.18.96.194:40489)
0501 02:02:50.424814 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:50.424844 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:50.425581 (+ 737us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:50.426125 (+ 544us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:50.426132 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:50.426454 (+ 322us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:50.427350 (+ 896us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.427363 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.428117 (+ 754us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.428120 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:50.428458 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.428463 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.428648 (+ 185us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.429192 (+ 544us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:50.429208 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:50.430906 (+ 1698us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:50.432998 (+ 2092us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.433002 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.433009 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.433209 (+ 200us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.433495 (+ 286us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.433501 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.433505 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.433628 (+ 123us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.433995 (+ 367us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:50.434002 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:50.434247 (+ 245us) client_negotiation.cc:770] Sending connection context
0501 02:02:50.434464 (+ 217us) client_negotiation.cc:241] Negotiation successful
0501 02:02:50.434705 (+ 241us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260501 02:02:50.436491 23338 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44833
I20260501 02:02:50.436714 23338 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:50.437161 23338 heartbeater.cc:507] Master 127.18.96.254:44833 requested a full tablet report, sending...
I20260501 02:02:50.438140 23009 ts_manager.cc:194] Registered new tserver with Master: 873fff57d063443198dd7f73ca79be75 (127.18.96.194:39759)
I20260501 02:02:50.438668 23009 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:40489
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:50.475129 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44833
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:50.571882 23345 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:50.572067 23345 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:50.572160 23345 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:50.575434 23345 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:50.575515 23345 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:50.575639 23345 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:50.579646 23345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:50.580499 23345 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:50.581260 23345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:50.587344 23353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:50.587349 23350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:50.587349 23351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:50.587563 23345 server_base.cc:1061] running on GCE node
I20260501 02:02:50.587932 23345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:50.588449 23345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:50.589625 23345 hybrid_clock.cc:648] HybridClock initialized: now 1777600970589610 us; error 47 us; skew 500 ppm
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:50.592090 23345 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:50.592959 23345 webserver.cc:492] Webserver started at http://127.18.96.195:45237/ using document root <none> and password file <none>
I20260501 02:02:50.593405 23345 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:50.593447 23345 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:50.593587 23345 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:50.595202 23345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "18465049a6f14f1994ecd65e75fc8bc5"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "df97c779ff19e3a5d56e8cfd451d938d"
server_key_iv: "fd007f94cb74741e0dd1289973673352"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.595611 23345 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "18465049a6f14f1994ecd65e75fc8bc5"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "df97c779ff19e3a5d56e8cfd451d938d"
server_key_iv: "fd007f94cb74741e0dd1289973673352"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.598657 23345 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260501 02:02:50.600525 23360 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.601436 23345 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:50.601563 23345 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "18465049a6f14f1994ecd65e75fc8bc5"
format_stamp: "Formatted at 2026-05-01 02:02:50 on dist-test-slave-2j90"
server_key: "df97c779ff19e3a5d56e8cfd451d938d"
server_key_iv: "fd007f94cb74741e0dd1289973673352"
server_key_version: "encryptionkey@0"
I20260501 02:02:50.601657 23345 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:50.626924 23345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:50.629283 23345 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:50.629492 23345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:50.630069 23345 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:50.630941 23345 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:50.630987 23345 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.631057 23345 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:50.631088 23345 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:50.639744 23345 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:44461
I20260501 02:02:50.639817 23473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:44461 every 8 connection(s)
I20260501 02:02:50.640638 23345 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:50.649868 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23345
I20260501 02:02:50.650019 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:50.650285 18819 external_mini_cluster.cc:1468] Setting key f5bded53d533c98fff44a6d76f37b9a7
I20260501 02:02:50.651990 23048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.642240 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:45069 (local address 127.18.96.254:44833)
0501 02:02:50.642422 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:50.642427 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:50.643015 (+ 588us) server_negotiation.cc:408] Connection header received
0501 02:02:50.643738 (+ 723us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:50.643741 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:50.643787 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:50.643865 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:50.645306 (+ 1441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.645791 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.646375 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.646499 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.648555 (+ 2056us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:50.648581 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:50.648583 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:50.648608 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:50.649887 (+ 1279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.650428 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.650434 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.650436 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.650487 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.650823 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.650826 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.650828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.651042 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:50.651164 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:50.651720 (+ 556us) server_negotiation.cc:300] Negotiation successful
0501 02:02:50.651828 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260501 02:02:50.652501 23476 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.642507 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44833 (local address 127.18.96.195:45069)
0501 02:02:50.642885 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:50.642915 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:50.643584 (+ 669us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:50.644010 (+ 426us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:50.644016 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:50.644376 (+ 360us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:50.645182 (+ 806us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.645194 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.645898 (+ 704us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.645900 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:50.646285 (+ 385us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:50.646291 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.646423 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.646978 (+ 555us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:50.646994 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:50.648409 (+ 1415us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:50.650047 (+ 1638us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.650073 (+ 26us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.650083 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.650307 (+ 224us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.650580 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:50.650584 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:50.650586 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:50.650720 (+ 134us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:50.651201 (+ 481us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:50.651206 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:50.651432 (+ 226us) client_negotiation.cc:770] Sending connection context
0501 02:02:50.651664 (+ 232us) client_negotiation.cc:241] Negotiation successful
0501 02:02:50.651840 (+ 176us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260501 02:02:50.653407 23474 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44833
I20260501 02:02:50.653625 23474 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:50.654140 23474 heartbeater.cc:507] Master 127.18.96.254:44833 requested a full tablet report, sending...
I20260501 02:02:50.655043 23009 ts_manager.cc:194] Registered new tserver with Master: 18465049a6f14f1994ecd65e75fc8bc5 (127.18.96.195:44461)
I20260501 02:02:50.655525 23009 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:45069
I20260501 02:02:50.663738 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:02:50.687150 23481 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.678509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.18.96.193:44637)
0501 02:02:50.678849 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:50.678854 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:50.678875 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:02:50.678927 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:50.678931 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:50.679078 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:50.679198 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:50.679927 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.680805 (+ 878us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.681759 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.682065 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.683960 (+ 1895us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:50.683997 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:50.684008 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:50.684030 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:50.685461 (+ 1431us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.685891 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.685897 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.685898 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.685943 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.686234 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.686237 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.686239 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.686414 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:50.686624 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:50.686861 (+ 237us) server_negotiation.cc:300] Negotiation successful
0501 02:02:50.686998 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:50.696333 23048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:50.689610 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.18.96.254:44833)
0501 02:02:50.689747 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:50.689750 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:50.689763 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:02:50.689875 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:50.689877 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:50.689918 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:50.690070 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:50.690799 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.691265 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:50.691786 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:50.691923 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:50.693878 (+ 1955us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:50.693892 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:50.693894 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:50.693914 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:50.695111 (+ 1197us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.695436 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.695438 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.695440 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.695474 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:50.695651 (+ 177us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:50.695653 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:50.695654 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:50.695787 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:50.695891 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:50.696058 (+ 167us) server_negotiation.cc:300] Negotiation successful
0501 02:02:50.696138 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260501 02:02:50.696745 23009 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40986
I20260501 02:02:50.697326 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23073
I20260501 02:02:50.703387 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23209
I20260501 02:02:50.709163 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23345
I20260501 02:02:50.714260 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 22977
2026-05-01T02:02:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3055 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[23486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[23486](info): set up 2 sockets
May 01 02:02:50 dist-test-slave-2j90 krb5kdc[23486](info): commencing operation
krb5kdc: starting...
W20260501 02:02:52.686586 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:02:52 dist-test-slave-2j90 krb5kdc[23486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:52.824477 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39061
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:52.922569 23502 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:52.922761 23502 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:52.922816 23502 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:52.925907 23502 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:52.925977 23502 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:52.926003 23502 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:52.926024 23502 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:52.926043 23502 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:52.930136 23502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:39061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:52.930989 23502 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:52.931705 23502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:52.936601 23510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:52.936592 23508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:52.936604 23507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:52.936848 23502 server_base.cc:1061] running on GCE node
I20260501 02:02:52.937467 23502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:52.938396 23502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:52.939584 23502 hybrid_clock.cc:648] HybridClock initialized: now 1777600972939569 us; error 29 us; skew 500 ppm
May 01 02:02:52 dist-test-slave-2j90 krb5kdc[23486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600972, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:52.941932 23502 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:02:52.942639 23502 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:52.942729 23502 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:52.942934 23502 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:52.944416 23502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "4c3e19a50860409e8026da2efd6d7908"
format_stamp: "Formatted at 2026-05-01 02:02:52 on dist-test-slave-2j90"
server_key: "ea20591953045a040eee5d403f7efe5e"
server_key_iv: "8be5b1697cdcf391179450e75e1df179"
server_key_version: "encryptionkey@0"
I20260501 02:02:52.944890 23502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "4c3e19a50860409e8026da2efd6d7908"
format_stamp: "Formatted at 2026-05-01 02:02:52 on dist-test-slave-2j90"
server_key: "ea20591953045a040eee5d403f7efe5e"
server_key_iv: "8be5b1697cdcf391179450e75e1df179"
server_key_version: "encryptionkey@0"
I20260501 02:02:52.947876 23502 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:52.949762 23516 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:52.950644 23502 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:52.950779 23502 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "4c3e19a50860409e8026da2efd6d7908"
format_stamp: "Formatted at 2026-05-01 02:02:52 on dist-test-slave-2j90"
server_key: "ea20591953045a040eee5d403f7efe5e"
server_key_iv: "8be5b1697cdcf391179450e75e1df179"
server_key_version: "encryptionkey@0"
I20260501 02:02:52.950873 23502 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:52.970171 23502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:52.976807 23502 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:52.977035 23502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:52.984488 23568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:39061 every 8 connection(s)
I20260501 02:02:52.984485 23502 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:39061
I20260501 02:02:52.985141 23502 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:52.988019 23569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:52.989291 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23502
I20260501 02:02:52.989421 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:52.989655 18819 external_mini_cluster.cc:1468] Setting key c00a7333792e702e24c4776a1554d474
I20260501 02:02:52.993155 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: Bootstrap starting.
I20260501 02:02:52.995640 23569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:52.996322 23569 log.cc:826] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: Log is configured to *not* fsync() on all Append() calls
May 01 02:02:52 dist-test-slave-2j90 krb5kdc[23486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:52.998225 23569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: No bootstrap required, opened a new log
I20260501 02:02:53.000759 23569 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } }
I20260501 02:02:53.001066 23569 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:53.001163 23569 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3e19a50860409e8026da2efd6d7908, State: Initialized, Role: FOLLOWER
I20260501 02:02:53.001608 23569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } }
I20260501 02:02:53.001724 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:52.990878 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.18.96.254:39061)
0501 02:02:52.991267 (+ 389us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:52.991276 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:52.991308 (+ 32us) server_negotiation.cc:408] Connection header received
0501 02:02:52.991806 (+ 498us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:52.991821 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:52.992085 (+ 264us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:52.992383 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:52.993283 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:52.994083 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:52.995222 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:52.995449 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:52.997683 (+ 2234us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:52.997699 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:52.997710 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:52.997735 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:52.999439 (+ 1704us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:52.999861 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:52.999865 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:52.999869 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:52.999936 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.000163 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.000167 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.000169 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.000435 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:53.000614 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.000897 (+ 283us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.001146 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260501 02:02:53.001812 23569 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:53.001888 23569 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:53.001994 23569 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:53.002992 23569 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } }
I20260501 02:02:53.003307 23569 leader_election.cc:304] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c3e19a50860409e8026da2efd6d7908; no voters:
I20260501 02:02:53.003639 23569 leader_election.cc:290] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:53.003729 23574 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:53.003985 23574 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [term 1 LEADER]: Becoming Leader. State: Replica: 4c3e19a50860409e8026da2efd6d7908, State: Running, Role: LEADER
I20260501 02:02:53.004258 23574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } }
I20260501 02:02:53.004817 23569 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:53.005802 23576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c3e19a50860409e8026da2efd6d7908. Latest consensus state: current_term: 1 leader_uuid: "4c3e19a50860409e8026da2efd6d7908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } } }
I20260501 02:02:53.005803 23575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c3e19a50860409e8026da2efd6d7908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c3e19a50860409e8026da2efd6d7908" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39061 } } }
I20260501 02:02:53.005975 23575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:53.005975 23576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:53.006582 23583 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:53.009276 23583 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:53.014478 23583 catalog_manager.cc:1357] Generated new cluster ID: 7fd506df73cc427f8a6f3440372187df
I20260501 02:02:53.014542 23583 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:53.024734 23583 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:53.025705 23583 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:53.035666 23583 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: Generated new TSK 0
I20260501 02:02:53.036206 23583 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:53.087224 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39061
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:53.189821 23597 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:53.190019 23597 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:53.190114 23597 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:53.193255 23597 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:53.193306 23597 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:53.193444 23597 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:53.197448 23597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:53.198369 23597 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:53.199151 23597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:53.204613 23605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.204643 23603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.204608 23602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:53.204973 23597 server_base.cc:1061] running on GCE node
I20260501 02:02:53.205312 23597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:53.205857 23597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:53.207042 23597 hybrid_clock.cc:648] HybridClock initialized: now 1777600973207023 us; error 39 us; skew 500 ppm
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:53.209419 23597 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:02:53.210090 23597 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:53.210168 23597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:53.210353 23597 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:53.211885 23597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "0a3b59c1bb764d8881f223ccbb5a07fa"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "06117ec09afa269fea268d966939d23c"
server_key_iv: "e86d5f0e17f267251c386fcd1526e22a"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.212343 23597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "0a3b59c1bb764d8881f223ccbb5a07fa"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "06117ec09afa269fea268d966939d23c"
server_key_iv: "e86d5f0e17f267251c386fcd1526e22a"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.215390 23597 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:02:53.217304 23611 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.218204 23597 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:53.218318 23597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "0a3b59c1bb764d8881f223ccbb5a07fa"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "06117ec09afa269fea268d966939d23c"
server_key_iv: "e86d5f0e17f267251c386fcd1526e22a"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.218429 23597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:53.242522 23597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:53.245071 23597 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:53.245316 23597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:53.245893 23597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:53.246850 23597 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:53.246927 23597 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.246999 23597 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:53.247054 23597 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.257591 23597 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:41745
I20260501 02:02:53.257632 23724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:41745 every 8 connection(s)
I20260501 02:02:53.258244 23597 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:53.261897 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23597
I20260501 02:02:53.262033 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:53.262298 18819 external_mini_cluster.cc:1468] Setting key 2c3b54eab0d00cb5c00ca7bc4313f816
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:02:53.271785 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.259882 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:50661 (local address 127.18.96.254:39061)
0501 02:02:53.260069 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.260074 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.260811 (+ 737us) server_negotiation.cc:408] Connection header received
0501 02:02:53.262532 (+ 1721us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.262536 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.262598 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.262688 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:53.264246 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.264776 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.265597 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.265841 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.267837 (+ 1996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:53.267852 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:53.267854 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:53.267885 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:53.269563 (+ 1678us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.270227 (+ 664us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.270230 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.270231 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.270275 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.270647 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.270649 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.270650 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.270792 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:53.270866 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.271521 (+ 655us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.271647 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:02:53.272336 23727 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.260178 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39061 (local address 127.18.96.193:50661)
0501 02:02:53.260673 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.260714 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.262279 (+ 1565us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.262872 (+ 593us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.262882 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.263280 (+ 398us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:53.263920 (+ 640us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.263936 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.264918 (+ 982us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.264924 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.265454 (+ 530us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.265462 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.265612 (+ 150us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.266229 (+ 617us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:53.266246 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:53.267671 (+ 1425us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:53.269706 (+ 2035us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.269714 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.269725 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.270103 (+ 378us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.270373 (+ 270us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.270377 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.270379 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.270543 (+ 164us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.270927 (+ 384us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:53.270935 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:53.271229 (+ 294us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.271435 (+ 206us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.271671 (+ 236us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260501 02:02:53.273473 23725 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39061
I20260501 02:02:53.273782 23725 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:02:53.274394 23725 heartbeater.cc:507] Master 127.18.96.254:39061 requested a full tablet report, sending...
I20260501 02:02:53.276010 23533 ts_manager.cc:194] Registered new tserver with Master: 0a3b59c1bb764d8881f223ccbb5a07fa (127.18.96.193:41745)
I20260501 02:02:53.277159 23533 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:50661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:53.310482 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39061
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:53.408470 23732 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:53.408696 23732 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:53.408789 23732 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:53.411962 23732 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:53.412062 23732 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:53.412185 23732 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:53.416376 23732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:53.417353 23732 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:53.418195 23732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:53.424134 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.424106 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.424106 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:53.424726 23732 server_base.cc:1061] running on GCE node
I20260501 02:02:53.425055 23732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:53.425557 23732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:53.426724 23732 hybrid_clock.cc:648] HybridClock initialized: now 1777600973426707 us; error 40 us; skew 500 ppm
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:53.429078 23732 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:02:53.429733 23732 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:53.429812 23732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:53.430073 23732 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:53.431612 23732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "ffd6d659759043c29d8fee8d958512b9"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "807752b619758bed893d21b1d84c20d5"
server_key_iv: "2387984c01db90e8aa8802251e12f259"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.432086 23732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "ffd6d659759043c29d8fee8d958512b9"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "807752b619758bed893d21b1d84c20d5"
server_key_iv: "2387984c01db90e8aa8802251e12f259"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.435230 23732 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:02:53.437220 23746 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.438131 23732 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:02:53.438244 23732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "ffd6d659759043c29d8fee8d958512b9"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "807752b619758bed893d21b1d84c20d5"
server_key_iv: "2387984c01db90e8aa8802251e12f259"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.438333 23732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:53.477432 23732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:53.479856 23732 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:53.480048 23732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:53.480556 23732 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:53.481391 23732 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:53.481460 23732 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.481528 23732 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:53.481580 23732 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.490811 23732 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:43379
I20260501 02:02:53.490849 23859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:43379 every 8 connection(s)
I20260501 02:02:53.491340 23732 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:53.495651 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23732
I20260501 02:02:53.495769 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:53.496039 18819 external_mini_cluster.cc:1468] Setting key aa5d789c335fa1c7a3170b9bf2660aff
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:53.503116 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.492862 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:43297 (local address 127.18.96.254:39061)
0501 02:02:53.493001 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.493005 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.493568 (+ 563us) server_negotiation.cc:408] Connection header received
0501 02:02:53.494939 (+ 1371us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.494943 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.495007 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.495103 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:53.496500 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.497044 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.497796 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.498031 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.499919 (+ 1888us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:53.499937 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:53.499942 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:53.499971 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:53.501340 (+ 1369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.501802 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.501807 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.501809 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.501859 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.502181 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.502184 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.502186 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.502319 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:53.502421 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.502850 (+ 429us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.502962 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:02:53.503628 23862 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.493086 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39061 (local address 127.18.96.194:43297)
0501 02:02:53.493449 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.493478 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.494721 (+ 1243us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.495253 (+ 532us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.495261 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.495716 (+ 455us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:53.496314 (+ 598us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.496330 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.497167 (+ 837us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.497171 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.497667 (+ 496us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.497675 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.497815 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.498358 (+ 543us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:53.498374 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:53.499782 (+ 1408us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:53.501455 (+ 1673us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.501462 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.501472 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.501698 (+ 226us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.501969 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.501975 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.501977 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.502077 (+ 100us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.502421 (+ 344us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:53.502428 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:53.502621 (+ 193us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.502801 (+ 180us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.503006 (+ 205us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:02:53.504644 23860 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39061
I20260501 02:02:53.504892 23860 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:53.505330 23860 heartbeater.cc:507] Master 127.18.96.254:39061 requested a full tablet report, sending...
I20260501 02:02:53.506796 23533 ts_manager.cc:194] Registered new tserver with Master: ffd6d659759043c29d8fee8d958512b9 (127.18.96.194:43379)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:02:53.507735 23533 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:43297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:02:53.541883 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39061
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:02:53.638331 23867 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:53.638516 23867 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:53.638609 23867 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:53.641781 23867 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:53.641834 23867 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:53.642014 23867 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:53.646117 23867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:53.647053 23867 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:53.647794 23867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:53.653446 23873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.653503 23872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:53.653474 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:53.653716 23867 server_base.cc:1061] running on GCE node
I20260501 02:02:53.654111 23867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:53.654646 23867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:53.655810 23867 hybrid_clock.cc:648] HybridClock initialized: now 1777600973655801 us; error 34 us; skew 500 ppm
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:02:53.658636 23867 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:02:53.659247 23867 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:53.659298 23867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:53.659438 23867 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:53.660899 23867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "a8906e92946643248227308f92860ef6"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "04bad6a6c9e98322cf143c34b1a31df3"
server_key_iv: "316965f8c522355dba96f898cf7e8867"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.661315 23867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "a8906e92946643248227308f92860ef6"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "04bad6a6c9e98322cf143c34b1a31df3"
server_key_iv: "316965f8c522355dba96f898cf7e8867"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.664299 23867 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:02:53.666232 23881 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.667151 23867 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260501 02:02:53.667256 23867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "a8906e92946643248227308f92860ef6"
format_stamp: "Formatted at 2026-05-01 02:02:53 on dist-test-slave-2j90"
server_key: "04bad6a6c9e98322cf143c34b1a31df3"
server_key_iv: "316965f8c522355dba96f898cf7e8867"
server_key_version: "encryptionkey@0"
I20260501 02:02:53.667325 23867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:53.682444 23867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:53.684494 23867 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:53.684623 23867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:53.685051 23867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:53.685787 23867 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:53.685833 23867 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.685868 23867 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:53.685883 23867 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:53.694450 23867 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:38183
I20260501 02:02:53.694495 23994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:38183 every 8 connection(s)
I20260501 02:02:53.694968 23867 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:53.696683 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 23867
I20260501 02:02:53.696825 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:53.697150 18819 external_mini_cluster.cc:1468] Setting key 2e90fc8ce3c3a908e53e161e9b8937d9
May 01 02:02:53 dist-test-slave-2j90 krb5kdc[23486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600973, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:02:53.706197 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.696480 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:54165 (local address 127.18.96.254:39061)
0501 02:02:53.696654 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.696679 (+ 25us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.697353 (+ 674us) server_negotiation.cc:408] Connection header received
0501 02:02:53.698961 (+ 1608us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.698964 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.699019 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.699105 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:53.700165 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.700631 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.701234 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.701355 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.703367 (+ 2012us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:53.703381 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:53.703383 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:53.703406 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:53.704552 (+ 1146us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.704970 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.704973 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.704974 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.705013 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.705250 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.705252 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.705253 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.705456 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:53.705529 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.705989 (+ 460us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.706085 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260501 02:02:53.706713 23997 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.696700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39061 (local address 127.18.96.195:54165)
0501 02:02:53.697197 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.697238 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.698784 (+ 1546us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.699275 (+ 491us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.699286 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.699624 (+ 338us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:53.700041 (+ 417us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.700052 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.700736 (+ 684us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.700738 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.701137 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.701143 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.701283 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.701812 (+ 529us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:02:53.701829 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:53.703242 (+ 1413us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:53.704651 (+ 1409us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.704670 (+ 19us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.704678 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.704886 (+ 208us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.705089 (+ 203us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:02:53.705091 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:02:53.705093 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:02:53.705178 (+ 85us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:02:53.705544 (+ 366us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:53.705548 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:53.705749 (+ 201us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.705907 (+ 158us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.706115 (+ 208us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":149,"threads_started":1}
I20260501 02:02:53.707708 23995 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39061
I20260501 02:02:53.707917 23995 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:53.708302 23995 heartbeater.cc:507] Master 127.18.96.254:39061 requested a full tablet report, sending...
I20260501 02:02:53.709172 23533 ts_manager.cc:194] Registered new tserver with Master: a8906e92946643248227308f92860ef6 (127.18.96.195:38183)
I20260501 02:02:53.709626 23533 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:54165
I20260501 02:02:53.710826 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:02:53.718572 23572 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.712668 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.18.96.254:39061)
0501 02:02:53.712800 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.712806 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.712931 (+ 125us) server_negotiation.cc:408] Connection header received
0501 02:02:53.713028 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.713031 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.713065 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.713121 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:53.713811 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.714302 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.714892 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.715012 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.715960 (+ 948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:53.715972 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:53.715974 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:02:53.715994 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:53.717292 (+ 1298us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.717608 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.717610 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.717612 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.717647 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:02:53.717845 (+ 198us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:02:53.717847 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:02:53.717848 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:02:53.718037 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:53.718135 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.718304 (+ 169us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.718381 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:02:53.720805 23532 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:53.723140 23532 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:53.735174 24008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.730489 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38183 (local address 127.0.0.1:54718)
0501 02:02:53.731058 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.731095 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.731215 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.731965 (+ 750us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.731973 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.731997 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:53.732296 (+ 299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.732303 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.733788 (+ 1485us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.733792 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.734824 (+ 1032us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.734831 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.734902 (+ 71us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.734926 (+ 24us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.734963 (+ 37us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.735012 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":86,"threads_started":1}
I20260501 02:02:53.735177 24009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.730635 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:41745 (local address 127.0.0.1:43558)
0501 02:02:53.731846 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.731861 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.731985 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.732349 (+ 364us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.732352 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.732369 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:53.732590 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.732596 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.733774 (+ 1178us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.733780 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.734638 (+ 858us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.734646 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.734759 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.734808 (+ 49us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.734895 (+ 87us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.734974 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"thread_start_us":106,"threads_started":1}
I20260501 02:02:53.735755 24011 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.731523 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.18.96.193:41745)
0501 02:02:53.731835 (+ 312us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.731841 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.732013 (+ 172us) server_negotiation.cc:408] Connection header received
0501 02:02:53.732121 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.732124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.732233 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.732365 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:53.732724 (+ 359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.733649 (+ 925us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.734760 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.735307 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.735435 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.735503 (+ 68us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.735597 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260501 02:02:53.736218 24006 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.731148 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:43379 (local address 127.0.0.1:54408)
0501 02:02:53.732059 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.732072 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.732157 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.732790 (+ 633us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.732794 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.732818 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:53.733049 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.733054 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.734684 (+ 1630us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.734688 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.735848 (+ 1160us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.735858 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.735995 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.736008 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.736059 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.736107 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":84,"threads_started":1}
I20260501 02:02:53.736399 24007 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.730570 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.18.96.195:38183)
0501 02:02:53.731489 (+ 919us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.731495 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.731516 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:02:53.731592 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.731600 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.731781 (+ 181us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.732431 (+ 650us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:53.732556 (+ 125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.733653 (+ 1097us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.735482 (+ 1829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.736010 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.736112 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.736165 (+ 53us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.736224 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":139,"threads_started":1}
I20260501 02:02:53.736860 24010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.731301 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.18.96.194:43379)
0501 02:02:53.732395 (+ 1094us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.732401 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.732419 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:02:53.732490 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.732495 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.732640 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.732782 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:53.733188 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.734546 (+ 1358us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.735996 (+ 1450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.736552 (+ 556us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.736635 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.736692 (+ 57us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.736755 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":110,"threads_started":1}
I20260501 02:02:53.739118 23659 tablet_service.cc:1511] Processing CreateTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d (DEFAULT_TABLE table=test-table [id=05ce126d72bc4d9cb0d24eb3429f9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:53.739122 23794 tablet_service.cc:1511] Processing CreateTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d (DEFAULT_TABLE table=test-table [id=05ce126d72bc4d9cb0d24eb3429f9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:53.739122 23929 tablet_service.cc:1511] Processing CreateTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d (DEFAULT_TABLE table=test-table [id=05ce126d72bc4d9cb0d24eb3429f9129]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:53.739938 23659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2d9ce1070b4f23acd52aa0ff05629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:53.739953 23929 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2d9ce1070b4f23acd52aa0ff05629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:53.739953 23794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca2d9ce1070b4f23acd52aa0ff05629d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:53.744973 24012 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Bootstrap starting.
I20260501 02:02:53.746345 24013 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Bootstrap starting.
I20260501 02:02:53.747187 24012 tablet_bootstrap.cc:654] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:53.747833 24012 log.cc:826] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:53.748076 24013 tablet_bootstrap.cc:654] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:53.748467 24014 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Bootstrap starting.
I20260501 02:02:53.748709 24013 log.cc:826] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:53.750093 24012 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: No bootstrap required, opened a new log
I20260501 02:02:53.750278 24012 ts_tablet_manager.cc:1403] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:02:53.750463 24013 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: No bootstrap required, opened a new log
I20260501 02:02:53.750560 24014 tablet_bootstrap.cc:654] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:53.750634 24013 ts_tablet_manager.cc:1403] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:02:53.751271 24014 log.cc:826] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:53.752920 24014 tablet_bootstrap.cc:492] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: No bootstrap required, opened a new log
I20260501 02:02:53.753193 24014 ts_tablet_manager.cc:1403] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:02:53.753937 24012 raft_consensus.cc:359] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.754060 24013 raft_consensus.cc:359] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.754172 24012 raft_consensus.cc:385] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:53.754201 24013 raft_consensus.cc:385] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:53.754232 24012 raft_consensus.cc:740] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3b59c1bb764d8881f223ccbb5a07fa, State: Initialized, Role: FOLLOWER
I20260501 02:02:53.754251 24013 raft_consensus.cc:740] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8906e92946643248227308f92860ef6, State: Initialized, Role: FOLLOWER
I20260501 02:02:53.754644 24012 consensus_queue.cc:260] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.754644 24013 consensus_queue.cc:260] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.755388 23725 heartbeater.cc:499] Master 127.18.96.254:39061 was elected leader, sending a full tablet report...
I20260501 02:02:53.755584 24012 ts_tablet_manager.cc:1434] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260501 02:02:53.755667 23995 heartbeater.cc:499] Master 127.18.96.254:39061 was elected leader, sending a full tablet report...
I20260501 02:02:53.756775 24013 ts_tablet_manager.cc:1434] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260501 02:02:53.756752 24014 raft_consensus.cc:359] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.756946 24014 raft_consensus.cc:385] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:53.757007 24014 raft_consensus.cc:740] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd6d659759043c29d8fee8d958512b9, State: Initialized, Role: FOLLOWER
I20260501 02:02:53.757396 24014 consensus_queue.cc:260] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.758080 23860 heartbeater.cc:499] Master 127.18.96.254:39061 was elected leader, sending a full tablet report...
I20260501 02:02:53.758330 24014 ts_tablet_manager.cc:1434] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260501 02:02:53.759212 23726 tablet.cc:2404] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:53.894040 24018 raft_consensus.cc:493] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:53.894271 24018 raft_consensus.cc:515] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.895368 24018 leader_election.cc:290] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8906e92946643248227308f92860ef6 (127.18.96.195:38183), ffd6d659759043c29d8fee8d958512b9 (127.18.96.194:43379)
I20260501 02:02:53.898689 24022 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.895617 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:43379 (local address 127.18.96.193:40911)
0501 02:02:53.896216 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.896230 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.896334 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.896572 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.896574 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.896589 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:53.896853 (+ 264us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.896860 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.897588 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.897591 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.898255 (+ 664us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.898281 (+ 26us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.898402 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.898417 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.898465 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.898514 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":11,"thread_start_us":98,"threads_started":1}
I20260501 02:02:53.898703 24021 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.895617 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38183 (local address 127.18.96.193:39493)
0501 02:02:53.896000 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.896017 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:53.896124 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:53.896395 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:53.896399 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:53.896419 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:02:53.896720 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.896727 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.897520 (+ 793us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.897527 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:53.898207 (+ 680us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:53.898215 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.898334 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.898374 (+ 40us) client_negotiation.cc:770] Sending connection context
0501 02:02:53.898426 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:02:53.898479 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":136,"threads_started":1}
I20260501 02:02:53.899108 24007 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.895787 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39493 (local address 127.18.96.195:38183)
0501 02:02:53.895944 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.895947 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.896106 (+ 159us) server_negotiation.cc:408] Connection header received
0501 02:02:53.896211 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.896215 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.896283 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.896364 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:53.896847 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.897404 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.898339 (+ 935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.898849 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.898892 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.898951 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.899004 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:02:53.899171 24010 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.895948 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:40911 (local address 127.18.96.194:43379)
0501 02:02:53.896091 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.896098 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.896320 (+ 222us) server_negotiation.cc:408] Connection header received
0501 02:02:53.896422 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.896425 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.896472 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.896557 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:02:53.896978 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.897480 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.898411 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.898883 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.898941 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.898995 (+ 54us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.899056 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:02:53.899684 23949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca2d9ce1070b4f23acd52aa0ff05629d" candidate_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8906e92946643248227308f92860ef6" is_pre_election: true
I20260501 02:02:53.899684 23814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca2d9ce1070b4f23acd52aa0ff05629d" candidate_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd6d659759043c29d8fee8d958512b9" is_pre_election: true
I20260501 02:02:53.899926 23814 raft_consensus.cc:2468] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3b59c1bb764d8881f223ccbb5a07fa in term 0.
I20260501 02:02:53.899957 23949 raft_consensus.cc:2468] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3b59c1bb764d8881f223ccbb5a07fa in term 0.
I20260501 02:02:53.900394 23613 leader_election.cc:304] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3b59c1bb764d8881f223ccbb5a07fa, a8906e92946643248227308f92860ef6; no voters:
I20260501 02:02:53.900616 24018 raft_consensus.cc:2804] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:02:53.900728 24018 raft_consensus.cc:493] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:53.900786 24018 raft_consensus.cc:3060] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:53.901680 24018 raft_consensus.cc:515] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.902050 24018 leader_election.cc:290] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [CANDIDATE]: Term 1 election: Requested vote from peers a8906e92946643248227308f92860ef6 (127.18.96.195:38183), ffd6d659759043c29d8fee8d958512b9 (127.18.96.194:43379)
I20260501 02:02:53.902379 23949 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca2d9ce1070b4f23acd52aa0ff05629d" candidate_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8906e92946643248227308f92860ef6"
I20260501 02:02:53.902403 23814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca2d9ce1070b4f23acd52aa0ff05629d" candidate_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd6d659759043c29d8fee8d958512b9"
I20260501 02:02:53.902473 23949 raft_consensus.cc:3060] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:53.902520 23814 raft_consensus.cc:3060] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:53.903436 23814 raft_consensus.cc:2468] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3b59c1bb764d8881f223ccbb5a07fa in term 1.
I20260501 02:02:53.903450 23949 raft_consensus.cc:2468] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3b59c1bb764d8881f223ccbb5a07fa in term 1.
I20260501 02:02:53.903750 23615 leader_election.cc:304] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3b59c1bb764d8881f223ccbb5a07fa, ffd6d659759043c29d8fee8d958512b9; no voters:
I20260501 02:02:53.903936 24018 raft_consensus.cc:2804] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:53.904207 24018 raft_consensus.cc:697] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [term 1 LEADER]: Becoming Leader. State: Replica: 0a3b59c1bb764d8881f223ccbb5a07fa, State: Running, Role: LEADER
I20260501 02:02:53.904484 24018 consensus_queue.cc:237] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } }
I20260501 02:02:53.907402 23533 catalog_manager.cc:5671] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a3b59c1bb764d8881f223ccbb5a07fa (127.18.96.193). New cstate: current_term: 1 leader_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3b59c1bb764d8881f223ccbb5a07fa" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 } health_report { overall_health: UNKNOWN } } }
I20260501 02:02:53.923230 24011 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.920145 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.18.96.193:41745)
0501 02:02:53.920259 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:53.920264 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:53.920296 (+ 32us) server_negotiation.cc:408] Connection header received
0501 02:02:53.920463 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:53.920465 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:53.920510 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:53.920622 (+ 112us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:02:53.920969 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.921449 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:53.922227 (+ 778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:53.922391 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:53.922463 (+ 72us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:02:53.922822 (+ 359us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:02:53.922908 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:53.923017 (+ 109us) server_negotiation.cc:300] Negotiation successful
0501 02:02:53.923056 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:02:53.927933 23814 raft_consensus.cc:1275] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 1 FOLLOWER]: Refusing update from remote peer 0a3b59c1bb764d8881f223ccbb5a07fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:53.927939 23949 raft_consensus.cc:1275] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 1 FOLLOWER]: Refusing update from remote peer 0a3b59c1bb764d8881f223ccbb5a07fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:53.928579 24023 consensus_queue.cc:1048] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8906e92946643248227308f92860ef6" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:53.928771 24018 consensus_queue.cc:1048] T ca2d9ce1070b4f23acd52aa0ff05629d P 0a3b59c1bb764d8881f223ccbb5a07fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd6d659759043c29d8fee8d958512b9" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:53.936338 24028 mvcc.cc:204] Tried to move back new op lower bound from 7281053589202735104 to 7281053589115199488. Current Snapshot: MvccSnapshot[applied={T|T < 7281053589202735104}]
I20260501 02:02:53.942155 23531 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:53.942302 23531 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:53.944586 23531 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4c3e19a50860409e8026da2efd6d7908: Sending DeleteTablet for 3 replicas of tablet ca2d9ce1070b4f23acd52aa0ff05629d
I20260501 02:02:53.945266 23929 tablet_service.cc:1558] Processing DeleteTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:54718
I20260501 02:02:53.945266 23794 tablet_service.cc:1558] Processing DeleteTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:54408
I20260501 02:02:53.945922 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23597
I20260501 02:02:53.946063 23659 tablet_service.cc:1558] Processing DeleteTablet for tablet ca2d9ce1070b4f23acd52aa0ff05629d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:43558
I20260501 02:02:53.947084 24035 tablet_replica.cc:333] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: stopping tablet replica
I20260501 02:02:53.947302 24035 raft_consensus.cc:2243] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:53.947542 24035 raft_consensus.cc:2272] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:53.948186 24036 tablet_replica.cc:333] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: stopping tablet replica
I20260501 02:02:53.948392 24036 raft_consensus.cc:2243] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:53.948630 24036 raft_consensus.cc:2272] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:53.948746 24035 ts_tablet_manager.cc:1916] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:53.949796 24036 ts_tablet_manager.cc:1916] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:53.950704 24035 ts_tablet_manager.cc:1929] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:53.950802 24035 log.cc:1199] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/ca2d9ce1070b4f23acd52aa0ff05629d
I20260501 02:02:53.951131 24035 ts_tablet_manager.cc:1950] T ca2d9ce1070b4f23acd52aa0ff05629d P ffd6d659759043c29d8fee8d958512b9: Deleting consensus metadata
I20260501 02:02:53.952189 23520 catalog_manager.cc:5002] TS ffd6d659759043c29d8fee8d958512b9 (127.18.96.194:43379): tablet ca2d9ce1070b4f23acd52aa0ff05629d (table test-table [id=05ce126d72bc4d9cb0d24eb3429f9129]) successfully deleted
I20260501 02:02:53.952934 24036 ts_tablet_manager.cc:1929] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:53.953022 24036 log.cc:1199] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/ca2d9ce1070b4f23acd52aa0ff05629d
I20260501 02:02:53.953365 24036 ts_tablet_manager.cc:1950] T ca2d9ce1070b4f23acd52aa0ff05629d P a8906e92946643248227308f92860ef6: Deleting consensus metadata
W20260501 02:02:53.953552 23520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:41745 (error 108)
I20260501 02:02:53.953775 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23732
I20260501 02:02:53.955355 24006 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:53.955066 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:41745 (local address 127.0.0.1:43580)
0501 02:02:53.955226 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:53.955277 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260501 02:02:53.955497 23520 catalog_manager.cc:4729] TS 0a3b59c1bb764d8881f223ccbb5a07fa (127.18.96.193:41745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca2d9ce1070b4f23acd52aa0ff05629d: Network error: Client connection negotiation failed: client connection to 127.18.96.193:41745: connect: Connection refused (error 111)
I20260501 02:02:53.955989 23518 catalog_manager.cc:5002] TS a8906e92946643248227308f92860ef6 (127.18.96.195:38183): tablet ca2d9ce1070b4f23acd52aa0ff05629d (table test-table [id=05ce126d72bc4d9cb0d24eb3429f9129]) successfully deleted
I20260501 02:02:53.959550 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23867
I20260501 02:02:53.965394 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 23502
2026-05-01T02:02:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3250 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-05-01T02:02:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:53Z Disabled control of system clock
I20260501 02:02:53.995292 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46037
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260501 02:02:54.094071 24041 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:54.094247 24041 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:54.094292 24041 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:54.097460 24041 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:54.097507 24041 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:54.097524 24041 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:54.097543 24041 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:54.097559 24041 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:02:54.101197 24041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:46037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:54.102137 24041 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:54.103113 24041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:54.108428 24049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.108440 24046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.108510 24041 server_base.cc:1061] running on GCE node
W20260501 02:02:54.108451 24047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.109130 24041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:54.109926 24041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:54.111152 24041 hybrid_clock.cc:648] HybridClock initialized: now 1777600974111035 us; error 139 us; skew 500 ppm
I20260501 02:02:54.112810 24041 webserver.cc:492] Webserver started at http://127.18.96.254:45093/ using document root <none> and password file <none>
I20260501 02:02:54.113312 24041 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:54.113390 24041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:54.113620 24041 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:54.115232 24041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "9d2982d0859c4cabad00455e44bac521"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "ba0ad6eaf076234d32d3c40c559398db"
server_key_iv: "0e103f127215622fe9b7d7f006e6f399"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.115706 24041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "9d2982d0859c4cabad00455e44bac521"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "ba0ad6eaf076234d32d3c40c559398db"
server_key_iv: "0e103f127215622fe9b7d7f006e6f399"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.118670 24041 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260501 02:02:54.120678 24055 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.121505 24041 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:02:54.121631 24041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9d2982d0859c4cabad00455e44bac521"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "ba0ad6eaf076234d32d3c40c559398db"
server_key_iv: "0e103f127215622fe9b7d7f006e6f399"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.121730 24041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:54.126593 24041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:54.127154 24041 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:54.127334 24041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:54.133778 24041 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:46037
I20260501 02:02:54.133831 24107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:46037 every 8 connection(s)
I20260501 02:02:54.134791 24041 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:54.137423 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:54.139717 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24041
I20260501 02:02:54.139827 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:54.140082 18819 external_mini_cluster.cc:1468] Setting key 9020fcc0da5c096718f9ee267fb9b2f1
I20260501 02:02:54.144264 24108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: Bootstrap starting.
I20260501 02:02:54.145022 24111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.141279 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.18.96.254:46037)
0501 02:02:54.141835 (+ 556us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.141846 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.141884 (+ 38us) server_negotiation.cc:408] Connection header received
0501 02:02:54.142542 (+ 658us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.142565 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.142907 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.143260 (+ 353us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.144022 (+ 762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.144027 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.144038 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.144049 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.144084 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.144096 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.144238 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.144320 (+ 82us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.144427 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260501 02:02:54.146523 24108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:54.147192 24108 log.cc:826] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:54.148761 24108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: No bootstrap required, opened a new log
I20260501 02:02:54.151134 24108 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } }
I20260501 02:02:54.151321 24108 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:54.151401 24108 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2982d0859c4cabad00455e44bac521, State: Initialized, Role: FOLLOWER
I20260501 02:02:54.151815 24108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } }
I20260501 02:02:54.151968 24108 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:54.152046 24108 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:54.152130 24108 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:54.153026 24108 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } }
I20260501 02:02:54.153369 24108 leader_election.cc:304] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d2982d0859c4cabad00455e44bac521; no voters:
I20260501 02:02:54.153628 24108 leader_election.cc:290] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:54.153774 24113 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:54.154027 24113 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [term 1 LEADER]: Becoming Leader. State: Replica: 9d2982d0859c4cabad00455e44bac521, State: Running, Role: LEADER
I20260501 02:02:54.154284 24113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } }
I20260501 02:02:54.154693 24108 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:54.155696 24114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d2982d0859c4cabad00455e44bac521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } } }
I20260501 02:02:54.155831 24114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:54.156237 24122 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:54.156301 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d2982d0859c4cabad00455e44bac521. Latest consensus state: current_term: 1 leader_uuid: "9d2982d0859c4cabad00455e44bac521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d2982d0859c4cabad00455e44bac521" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46037 } } }
I20260501 02:02:54.156385 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521 [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:54.159236 24122 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:54.163947 24122 catalog_manager.cc:1357] Generated new cluster ID: ab5f1ebb74974f08a6947f5d6c6389b9
I20260501 02:02:54.164055 24122 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:54.193099 24122 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:54.193372 24122 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:54.207194 24122 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: Generated new TSK 0
I20260501 02:02:54.207762 24122 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:02:54.218063 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--builtin_ntp_servers=127.18.96.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260501 02:02:54.319561 24132 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:54.319764 24132 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:54.319836 24132 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:54.323137 24132 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:54.323208 24132 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:54.323309 24132 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:02:54.327257 24132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:54.328233 24132 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:54.329303 24132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:54.335182 24137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.335366 24132 server_base.cc:1061] running on GCE node
W20260501 02:02:54.335197 24138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.335197 24140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.335886 24132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:54.336351 24132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:54.337498 24132 hybrid_clock.cc:648] HybridClock initialized: now 1777600974337458 us; error 53 us; skew 500 ppm
I20260501 02:02:54.339135 24132 webserver.cc:492] Webserver started at http://127.18.96.193:37459/ using document root <none> and password file <none>
I20260501 02:02:54.339622 24132 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:54.339673 24132 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:54.339859 24132 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:54.341349 24132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "5b37b1f4e63a4b81b82c7399f1c5750b"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "6a6de0aa9058077f6b0448991f68e1f3"
server_key_iv: "37397b2c75d6851127931b20376a344f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.341805 24132 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "5b37b1f4e63a4b81b82c7399f1c5750b"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "6a6de0aa9058077f6b0448991f68e1f3"
server_key_iv: "37397b2c75d6851127931b20376a344f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.344893 24132 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260501 02:02:54.346930 24146 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.347819 24132 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:54.347960 24132 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "5b37b1f4e63a4b81b82c7399f1c5750b"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "6a6de0aa9058077f6b0448991f68e1f3"
server_key_iv: "37397b2c75d6851127931b20376a344f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.348052 24132 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:54.352273 24132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:54.352742 24132 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:54.352936 24132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:54.353386 24132 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:54.354292 24132 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:54.354360 24132 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.354421 24132 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:54.354453 24132 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.363081 24132 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:36223
I20260501 02:02:54.363153 24259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:36223 every 8 connection(s)
I20260501 02:02:54.363937 24132 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:02:54.368753 24111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.365575 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:41827 (local address 127.18.96.254:46037)
0501 02:02:54.365717 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.365721 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.366454 (+ 733us) server_negotiation.cc:408] Connection header received
0501 02:02:54.367204 (+ 750us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.367208 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.367252 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.367370 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.368289 (+ 919us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.368292 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.368295 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.368306 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.368334 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.368341 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.368435 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.368578 (+ 143us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.368614 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:02:54.369231 24262 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.365823 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46037 (local address 127.18.96.193:41827)
0501 02:02:54.366329 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.366356 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.367015 (+ 659us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.367530 (+ 515us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.367536 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.367586 (+ 50us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.368076 (+ 490us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.368077 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.368097 (+ 20us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.368134 (+ 37us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.368429 (+ 295us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.368436 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.368442 (+ 6us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.368613 (+ 171us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.368701 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
I20260501 02:02:54.370451 24260 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46037
I20260501 02:02:54.370612 24260 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:54.370925 24260 heartbeater.cc:507] Master 127.18.96.254:46037 requested a full tablet report, sending...
I20260501 02:02:54.372184 24066 ts_manager.cc:194] Registered new tserver with Master: 5b37b1f4e63a4b81b82c7399f1c5750b (127.18.96.193:36223)
I20260501 02:02:54.373271 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24132
I20260501 02:02:54.373399 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:02:54.373632 18819 external_mini_cluster.cc:1468] Setting key 4047ca80ba722d55412e62b33542cbd9
I20260501 02:02:54.375319 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--builtin_ntp_servers=127.18.96.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260501 02:02:54.472191 24263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:54.472364 24263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:54.472410 24263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:54.475688 24263 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:54.475734 24263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:54.475800 24263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:02:54.479451 24263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:54.480389 24263 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:54.481452 24263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:54.487648 24271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.487748 24268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.487715 24269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.487926 24263 server_base.cc:1061] running on GCE node
I20260501 02:02:54.488272 24263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:54.488749 24263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:54.489907 24263 hybrid_clock.cc:648] HybridClock initialized: now 1777600974489897 us; error 25 us; skew 500 ppm
I20260501 02:02:54.491562 24263 webserver.cc:492] Webserver started at http://127.18.96.194:43317/ using document root <none> and password file <none>
I20260501 02:02:54.492095 24263 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:54.492147 24263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:54.492363 24263 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:54.493840 24263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "98f2a40591f2488e862de98dae80d1bf"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "181021bcd73c7654ce52766b0891f56e"
server_key_iv: "5586e19fa4a5b13d34d30c647ed963df"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.494385 24263 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "98f2a40591f2488e862de98dae80d1bf"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "181021bcd73c7654ce52766b0891f56e"
server_key_iv: "5586e19fa4a5b13d34d30c647ed963df"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.497246 24263 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:02:54.499154 24277 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.500063 24263 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:54.500193 24263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "98f2a40591f2488e862de98dae80d1bf"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "181021bcd73c7654ce52766b0891f56e"
server_key_iv: "5586e19fa4a5b13d34d30c647ed963df"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.500286 24263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:54.504474 24263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:54.504962 24263 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:54.505125 24263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:54.505575 24263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:54.506472 24263 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:54.506539 24263 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.506600 24263 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:54.506661 24263 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.515072 24263 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:38903
I20260501 02:02:54.515100 24390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:38903 every 8 connection(s)
I20260501 02:02:54.515888 24263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:02:54.520236 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24263
I20260501 02:02:54.520345 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:02:54.520561 24111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.517464 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:59403 (local address 127.18.96.254:46037)
0501 02:02:54.517595 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.517599 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.518391 (+ 792us) server_negotiation.cc:408] Connection header received
0501 02:02:54.519069 (+ 678us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.519071 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.519109 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.519174 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.520003 (+ 829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.520005 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.520008 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.520016 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.520052 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.520066 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.520150 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.520367 (+ 217us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.520403 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:02:54.520602 18819 external_mini_cluster.cc:1468] Setting key 323a0b96fd165c7ee4785c4122bbdf44
I20260501 02:02:54.521466 24393 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.517700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46037 (local address 127.18.96.194:59403)
0501 02:02:54.518275 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.518302 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.518896 (+ 594us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.519305 (+ 409us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.519311 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.519362 (+ 51us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.519814 (+ 452us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.519815 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.519834 (+ 19us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.519873 (+ 39us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.520193 (+ 320us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.520201 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.520209 (+ 8us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.520780 (+ 571us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.520912 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260501 02:02:54.522720 24391 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46037
I20260501 02:02:54.522774 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--builtin_ntp_servers=127.18.96.212:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260501 02:02:54.522956 24391 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:54.523346 24391 heartbeater.cc:507] Master 127.18.96.254:46037 requested a full tablet report, sending...
I20260501 02:02:54.524402 24066 ts_manager.cc:194] Registered new tserver with Master: 98f2a40591f2488e862de98dae80d1bf (127.18.96.194:38903)
W20260501 02:02:54.620894 24394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:54.621070 24394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:54.621114 24394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:54.624248 24394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:54.624289 24394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:54.624351 24394 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:02:54.628127 24394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:54.628990 24394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:54.630041 24394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:54.635916 24399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.636032 24400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.636087 24394 server_base.cc:1061] running on GCE node
W20260501 02:02:54.635929 24402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.636446 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:54.636984 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:54.638137 24394 hybrid_clock.cc:648] HybridClock initialized: now 1777600974638117 us; error 30 us; skew 500 ppm
I20260501 02:02:54.639689 24394 webserver.cc:492] Webserver started at http://127.18.96.195:42649/ using document root <none> and password file <none>
I20260501 02:02:54.640213 24394 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:54.640265 24394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:54.640491 24394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:54.642160 24394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "5daef3e5311f4633939357a6fcbe1747"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "e46051210db47c254bd06e595d5a0618"
server_key_iv: "ab2cb013a44f7b8aa4d7989b3a98d05f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.642623 24394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "5daef3e5311f4633939357a6fcbe1747"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "e46051210db47c254bd06e595d5a0618"
server_key_iv: "ab2cb013a44f7b8aa4d7989b3a98d05f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.645569 24394 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:02:54.647639 24408 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.648566 24394 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:02:54.648687 24394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "5daef3e5311f4633939357a6fcbe1747"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "e46051210db47c254bd06e595d5a0618"
server_key_iv: "ab2cb013a44f7b8aa4d7989b3a98d05f"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.648788 24394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:54.652973 24394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:54.653468 24394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:54.653637 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:54.654191 24394 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:02:54.655076 24394 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:02:54.655154 24394 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.655216 24394 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:02:54.655259 24394 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:54.664125 24394 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:39943
I20260501 02:02:54.664151 24521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:39943 every 8 connection(s)
I20260501 02:02:54.664959 24394 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:02:54.668416 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24394
I20260501 02:02:54.668538 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:02:54.668792 18819 external_mini_cluster.cc:1468] Setting key ce4a7b0b279e560f61fa447377702c32
I20260501 02:02:54.670020 24111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.666614 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:41099 (local address 127.18.96.254:46037)
0501 02:02:54.666744 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.666747 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.667587 (+ 840us) server_negotiation.cc:408] Connection header received
0501 02:02:54.668281 (+ 694us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.668285 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.668334 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.668446 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.669478 (+ 1032us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.669482 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.669484 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.669500 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.669536 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.669546 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.669657 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.669818 (+ 161us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.669846 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:02:54.670662 24524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.666875 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:46037 (local address 127.18.96.195:41099)
0501 02:02:54.667460 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.667488 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.668112 (+ 624us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.668631 (+ 519us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.668638 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.668699 (+ 61us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.669282 (+ 583us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.669284 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.669316 (+ 32us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.669355 (+ 39us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.669645 (+ 290us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.669671 (+ 26us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.669677 (+ 6us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.669850 (+ 173us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.670031 (+ 181us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260501 02:02:54.671761 24522 heartbeater.cc:344] Connected to a master server at 127.18.96.254:46037
I20260501 02:02:54.671981 24522 heartbeater.cc:461] Registering TS with master...
I20260501 02:02:54.672333 24522 heartbeater.cc:507] Master 127.18.96.254:46037 requested a full tablet report, sending...
I20260501 02:02:54.673226 24066 ts_manager.cc:194] Registered new tserver with Master: 5daef3e5311f4633939357a6fcbe1747 (127.18.96.195:39943)
I20260501 02:02:54.682485 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:02:54.685698 24111 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.684211 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.18.96.254:46037)
0501 02:02:54.684355 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.684359 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.684476 (+ 117us) server_negotiation.cc:408] Connection header received
0501 02:02:54.684585 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.684587 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.684617 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.684674 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.685345 (+ 671us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.685348 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.685350 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.685356 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.685370 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.685376 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.685437 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.685495 (+ 58us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.685522 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260501 02:02:54.688088 24066 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:02:54.690421 24066 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:02:54.700587 24532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.698308 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:38903 (local address 127.0.0.1:60834)
0501 02:02:54.698953 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.698978 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.699067 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.699432 (+ 365us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.699435 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.699447 (+ 12us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.699945 (+ 498us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.699947 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.699964 (+ 17us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.700016 (+ 52us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.700304 (+ 288us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.700313 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.700320 (+ 7us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.700426 (+ 106us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.700465 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":64,"threads_started":1}
I20260501 02:02:54.700630 24533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.698462 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.18.96.194:38903)
0501 02:02:54.698813 (+ 351us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.698821 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.699075 (+ 254us) server_negotiation.cc:408] Connection header received
0501 02:02:54.699167 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.699172 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.699301 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.699437 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.700151 (+ 714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.700157 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.700167 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.700177 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.700214 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.700227 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.700338 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.700437 (+ 99us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.700498 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":94,"threads_started":1}
I20260501 02:02:54.700959 24531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.697790 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.18.96.195:39943)
0501 02:02:54.698165 (+ 375us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.698169 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.698212 (+ 43us) server_negotiation.cc:408] Connection header received
0501 02:02:54.698264 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.698268 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.698420 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.698550 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.700310 (+ 1760us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.700315 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.700322 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.700331 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.700363 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.700373 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.700480 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.700760 (+ 280us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.700823 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":167,"threads_started":1}
I20260501 02:02:54.701015 24535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.698929 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.18.96.193:36223)
0501 02:02:54.699298 (+ 369us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.699304 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.699323 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:02:54.699414 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.699420 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.699574 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.699753 (+ 179us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.700306 (+ 553us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.700311 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.700320 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.700330 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.700358 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.700369 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.700480 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.700821 (+ 341us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.700868 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":115,"threads_started":1}
I20260501 02:02:54.701220 24534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.698576 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:36223 (local address 127.0.0.1:47590)
0501 02:02:54.699252 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.699264 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.699325 (+ 61us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.699748 (+ 423us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.699751 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.699765 (+ 14us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.700155 (+ 390us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.700157 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.700167 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.700186 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.700521 (+ 335us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.700525 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.700529 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.700598 (+ 69us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.700638 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":7,"thread_start_us":45,"threads_started":1}
I20260501 02:02:54.701222 24530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.697657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:39943 (local address 127.0.0.1:41366)
0501 02:02:54.697992 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.698021 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.698108 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.699192 (+ 1084us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.699197 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.699250 (+ 53us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.700137 (+ 887us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.700139 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.700152 (+ 13us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.700174 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.700521 (+ 347us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.700525 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.700529 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.700598 (+ 69us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.700627 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260501 02:02:54.703763 24194 tablet_service.cc:1511] Processing CreateTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 (DEFAULT_TABLE table=test-table [id=21cc94ef9bab407c89fd50a4df608026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:54.703984 24456 tablet_service.cc:1511] Processing CreateTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 (DEFAULT_TABLE table=test-table [id=21cc94ef9bab407c89fd50a4df608026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:54.703984 24325 tablet_service.cc:1511] Processing CreateTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 (DEFAULT_TABLE table=test-table [id=21cc94ef9bab407c89fd50a4df608026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:02:54.704751 24194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9960e5b8b3049c8a932e473b9c508e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:54.704847 24456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9960e5b8b3049c8a932e473b9c508e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:54.704838 24325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9960e5b8b3049c8a932e473b9c508e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:54.710215 24536 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: Bootstrap starting.
I20260501 02:02:54.711043 24538 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Bootstrap starting.
I20260501 02:02:54.711043 24537 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Bootstrap starting.
I20260501 02:02:54.712278 24536 tablet_bootstrap.cc:654] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:54.712991 24537 tablet_bootstrap.cc:654] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:54.712991 24538 tablet_bootstrap.cc:654] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:54.713039 24536 log.cc:826] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:54.713721 24537 log.cc:826] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:54.713721 24538 log.cc:826] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:54.714804 24536 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: No bootstrap required, opened a new log
I20260501 02:02:54.714993 24536 ts_tablet_manager.cc:1403] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:02:54.715310 24538 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: No bootstrap required, opened a new log
I20260501 02:02:54.715413 24537 tablet_bootstrap.cc:492] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: No bootstrap required, opened a new log
I20260501 02:02:54.715513 24538 ts_tablet_manager.cc:1403] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260501 02:02:54.715581 24537 ts_tablet_manager.cc:1403] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:02:54.717630 24538 raft_consensus.cc:359] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.717830 24538 raft_consensus.cc:385] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:54.717885 24538 raft_consensus.cc:740] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f2a40591f2488e862de98dae80d1bf, State: Initialized, Role: FOLLOWER
I20260501 02:02:54.717996 24536 raft_consensus.cc:359] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.718183 24536 raft_consensus.cc:385] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:54.718245 24536 raft_consensus.cc:740] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b37b1f4e63a4b81b82c7399f1c5750b, State: Initialized, Role: FOLLOWER
I20260501 02:02:54.718344 24538 consensus_queue.cc:260] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.718423 24537 raft_consensus.cc:359] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.718600 24537 raft_consensus.cc:385] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:54.718670 24537 raft_consensus.cc:740] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5daef3e5311f4633939357a6fcbe1747, State: Initialized, Role: FOLLOWER
I20260501 02:02:54.719041 24537 consensus_queue.cc:260] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.719178 24538 ts_tablet_manager.cc:1434] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260501 02:02:54.719391 24391 heartbeater.cc:499] Master 127.18.96.254:46037 was elected leader, sending a full tablet report...
I20260501 02:02:54.719357 24536 consensus_queue.cc:260] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.719806 24537 ts_tablet_manager.cc:1434] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:02:54.720126 24536 ts_tablet_manager.cc:1434] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:02:54.720132 24522 heartbeater.cc:499] Master 127.18.96.254:46037 was elected leader, sending a full tablet report...
I20260501 02:02:54.720224 24260 heartbeater.cc:499] Master 127.18.96.254:46037 was elected leader, sending a full tablet report...
I20260501 02:02:54.761430 24543 raft_consensus.cc:493] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:02:54.761585 24543 raft_consensus.cc:515] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.762653 24543 leader_election.cc:290] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98f2a40591f2488e862de98dae80d1bf (127.18.96.194:38903), 5b37b1f4e63a4b81b82c7399f1c5750b (127.18.96.193:36223)
I20260501 02:02:54.764519 24524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.762849 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:38903 (local address 127.18.96.195:48059)
0501 02:02:54.762997 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.763011 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.763109 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.763397 (+ 288us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.763402 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.763417 (+ 15us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.764001 (+ 584us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.764003 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.764013 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.764032 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.764280 (+ 248us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.764284 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.764288 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.764376 (+ 88us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.764404 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260501 02:02:54.764524 24533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.762991 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:48059 (local address 127.18.96.194:38903)
0501 02:02:54.763131 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.763137 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.763153 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:02:54.763197 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.763202 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.763245 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.763634 (+ 389us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.764139 (+ 505us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.764142 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.764145 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.764155 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.764190 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.764197 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.764281 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.764390 (+ 109us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.764422 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:02:54.765182 24535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.763217 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:35645 (local address 127.18.96.193:36223)
0501 02:02:54.763376 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.763381 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.763518 (+ 137us) server_negotiation.cc:408] Connection header received
0501 02:02:54.763588 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.763595 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.763635 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.763740 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.764839 (+ 1099us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.764845 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.764850 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.764861 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.764887 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.764894 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.764971 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.765069 (+ 98us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.765095 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:02:54.765281 24545 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.763121 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:36223 (local address 127.18.96.195:35645)
0501 02:02:54.763432 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.763444 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:54.763519 (+ 75us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:54.763770 (+ 251us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:54.763774 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:54.763788 (+ 14us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:02:54.764699 (+ 911us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:02:54.764702 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:02:54.764717 (+ 15us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:02:54.764739 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:02:54.764987 (+ 248us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:02:54.764990 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:02:54.764993 (+ 3us) client_negotiation.cc:770] Sending connection context
0501 02:02:54.765075 (+ 82us) client_negotiation.cc:241] Negotiation successful
0501 02:02:54.765102 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
I20260501 02:02:54.765125 24345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9960e5b8b3049c8a932e473b9c508e3" candidate_uuid: "5daef3e5311f4633939357a6fcbe1747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f2a40591f2488e862de98dae80d1bf" is_pre_election: true
I20260501 02:02:54.765894 24345 raft_consensus.cc:2468] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5daef3e5311f4633939357a6fcbe1747 in term 0.
I20260501 02:02:54.766397 24410 leader_election.cc:304] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5daef3e5311f4633939357a6fcbe1747, 98f2a40591f2488e862de98dae80d1bf; no voters:
I20260501 02:02:54.765895 24214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9960e5b8b3049c8a932e473b9c508e3" candidate_uuid: "5daef3e5311f4633939357a6fcbe1747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" is_pre_election: true
I20260501 02:02:54.766644 24543 raft_consensus.cc:2804] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260501 02:02:54.766661 24392 tablet.cc:2404] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:02:54.766708 24543 raft_consensus.cc:493] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:02:54.766738 24543 raft_consensus.cc:3060] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:54.766727 24214 raft_consensus.cc:2468] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5daef3e5311f4633939357a6fcbe1747 in term 0.
I20260501 02:02:54.767817 24543 raft_consensus.cc:515] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.768129 24543 leader_election.cc:290] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [CANDIDATE]: Term 1 election: Requested vote from peers 98f2a40591f2488e862de98dae80d1bf (127.18.96.194:38903), 5b37b1f4e63a4b81b82c7399f1c5750b (127.18.96.193:36223)
I20260501 02:02:54.768517 24214 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9960e5b8b3049c8a932e473b9c508e3" candidate_uuid: "5daef3e5311f4633939357a6fcbe1747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b"
I20260501 02:02:54.768517 24345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9960e5b8b3049c8a932e473b9c508e3" candidate_uuid: "5daef3e5311f4633939357a6fcbe1747" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98f2a40591f2488e862de98dae80d1bf"
I20260501 02:02:54.768627 24214 raft_consensus.cc:3060] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:54.768627 24345 raft_consensus.cc:3060] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:54.769840 24214 raft_consensus.cc:2468] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5daef3e5311f4633939357a6fcbe1747 in term 1.
I20260501 02:02:54.769860 24345 raft_consensus.cc:2468] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5daef3e5311f4633939357a6fcbe1747 in term 1.
I20260501 02:02:54.770159 24410 leader_election.cc:304] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b37b1f4e63a4b81b82c7399f1c5750b, 5daef3e5311f4633939357a6fcbe1747; no voters:
I20260501 02:02:54.770310 24543 raft_consensus.cc:2804] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:54.770519 24543 raft_consensus.cc:697] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 1 LEADER]: Becoming Leader. State: Replica: 5daef3e5311f4633939357a6fcbe1747, State: Running, Role: LEADER
I20260501 02:02:54.770819 24543 consensus_queue.cc:237] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } }
I20260501 02:02:54.773810 24071 catalog_manager.cc:5671] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5daef3e5311f4633939357a6fcbe1747 (127.18.96.195). New cstate: current_term: 1 leader_uuid: "5daef3e5311f4633939357a6fcbe1747" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5daef3e5311f4633939357a6fcbe1747" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 39943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 } health_report { overall_health: UNKNOWN } } }
I20260501 02:02:54.796314 24531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.795035 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.18.96.195:39943)
0501 02:02:54.795170 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:54.795173 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:54.795186 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:02:54.795246 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:54.795249 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:54.795297 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:54.795363 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:54.795989 (+ 626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:54.795992 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:54.795994 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:54.796003 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:54.796021 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:54.796028 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:54.796096 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:54.796174 (+ 78us) server_negotiation.cc:300] Negotiation successful
0501 02:02:54.796201 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:02:54.801268 24214 raft_consensus.cc:1275] T c9960e5b8b3049c8a932e473b9c508e3 P 5b37b1f4e63a4b81b82c7399f1c5750b [term 1 FOLLOWER]: Refusing update from remote peer 5daef3e5311f4633939357a6fcbe1747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:54.801268 24345 raft_consensus.cc:1275] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 1 FOLLOWER]: Refusing update from remote peer 5daef3e5311f4633939357a6fcbe1747: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:02:54.801927 24546 consensus_queue.cc:1048] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f2a40591f2488e862de98dae80d1bf" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:54.802112 24543 consensus_queue.cc:1048] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b37b1f4e63a4b81b82c7399f1c5750b" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:02:54.808017 24549 mvcc.cc:204] Tried to move back new op lower bound from 7281053592779431936 to 7281053592664076288. Current Snapshot: MvccSnapshot[applied={T|T < 7281053592779431936}]
I20260501 02:02:54.808837 24551 mvcc.cc:204] Tried to move back new op lower bound from 7281053592779431936 to 7281053592664076288. Current Snapshot: MvccSnapshot[applied={T|T < 7281053592779431936}]
I20260501 02:02:54.809074 24550 mvcc.cc:204] Tried to move back new op lower bound from 7281053592779431936 to 7281053592664076288. Current Snapshot: MvccSnapshot[applied={T|T < 7281053592779431936}]
I20260501 02:02:54.815879 24071 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:54.816045 24071 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:02:54.818378 24071 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9d2982d0859c4cabad00455e44bac521: Sending DeleteTablet for 3 replicas of tablet c9960e5b8b3049c8a932e473b9c508e3
I20260501 02:02:54.819020 24456 tablet_service.cc:1558] Processing DeleteTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:54 UTC) from {username='slave'} at 127.0.0.1:41366
I20260501 02:02:54.819134 24325 tablet_service.cc:1558] Processing DeleteTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:54 UTC) from {username='slave'} at 127.0.0.1:60834
I20260501 02:02:54.819159 24194 tablet_service.cc:1558] Processing DeleteTablet for tablet c9960e5b8b3049c8a932e473b9c508e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:02:54 UTC) from {username='slave'} at 127.0.0.1:47590
I20260501 02:02:54.819626 24559 tablet_replica.cc:333] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: stopping tablet replica
I20260501 02:02:54.819849 24560 tablet_replica.cc:333] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: stopping tablet replica
I20260501 02:02:54.819911 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24132
I20260501 02:02:54.819902 24559 raft_consensus.cc:2243] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:02:54.820268 24559 raft_consensus.cc:2272] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:54.820781 24560 raft_consensus.cc:2243] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:02:54.821058 24560 raft_consensus.cc:2272] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:02:54.822798 24560 ts_tablet_manager.cc:1916] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:54.823444 24559 ts_tablet_manager.cc:1916] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:02:54.825277 24559 ts_tablet_manager.cc:1929] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:54.825299 24560 ts_tablet_manager.cc:1929] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:02:54.825379 24560 log.cc:1199] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/c9960e5b8b3049c8a932e473b9c508e3
I20260501 02:02:54.825381 24559 log.cc:1199] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/c9960e5b8b3049c8a932e473b9c508e3
I20260501 02:02:54.825672 24559 ts_tablet_manager.cc:1950] T c9960e5b8b3049c8a932e473b9c508e3 P 5daef3e5311f4633939357a6fcbe1747: Deleting consensus metadata
I20260501 02:02:54.825708 24560 ts_tablet_manager.cc:1950] T c9960e5b8b3049c8a932e473b9c508e3 P 98f2a40591f2488e862de98dae80d1bf: Deleting consensus metadata
I20260501 02:02:54.826752 24057 catalog_manager.cc:5002] TS 5daef3e5311f4633939357a6fcbe1747 (127.18.96.195:39943): tablet c9960e5b8b3049c8a932e473b9c508e3 (table test-table [id=21cc94ef9bab407c89fd50a4df608026]) successfully deleted
I20260501 02:02:54.826965 24057 catalog_manager.cc:5002] TS 98f2a40591f2488e862de98dae80d1bf (127.18.96.194:38903): tablet c9960e5b8b3049c8a932e473b9c508e3 (table test-table [id=21cc94ef9bab407c89fd50a4df608026]) successfully deleted
W20260501 02:02:54.831228 24057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:36223 (error 108)
I20260501 02:02:54.831418 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24263
I20260501 02:02:54.833595 24530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:54.833327 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:36223 (local address 127.0.0.1:47594)
0501 02:02:54.833467 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:54.833524 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260501 02:02:54.833783 24057 catalog_manager.cc:4729] TS 5b37b1f4e63a4b81b82c7399f1c5750b (127.18.96.193:36223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9960e5b8b3049c8a932e473b9c508e3: Network error: Client connection negotiation failed: client connection to 127.18.96.193:36223: connect: Connection refused (error 111)
I20260501 02:02:54.839136 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24394
I20260501 02:02:54.845201 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24041
2026-05-01T02:02:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (877 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260501 02:02:54.865453 18819 mini_oidc.cc:150] Starting JWKS server
I20260501 02:02:54.866922 18819 webserver.cc:492] Webserver started at https://127.0.0.1:35545/ using document root <none> and password file <none>
I20260501 02:02:54.867049 18819 mini_oidc.cc:186] Starting OIDC Discovery server
I20260501 02:02:54.867228 18819 webserver.cc:492] Webserver started at http://127.0.0.1:42251/ using document root <none> and password file <none>
2026-05-01T02:02:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:02:54Z Disabled control of system clock
I20260501 02:02:54.874974 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44587
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44587
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/testchainca.cert with env {}
W20260501 02:02:54.971824 24568 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:02:54.972004 24568 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:02:54.972051 24568 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:02:54.975224 24568 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:02:54.975273 24568 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:02:54.975289 24568 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:02:54.975307 24568 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:02:54.975324 24568 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260501 02:02:54.975342 24568 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260501 02:02:54.975355 24568 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:35545/jwks.json
I20260501 02:02:54.979452 24568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:02:54.980314 24568 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:02:54.981029 24568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:02:54.986366 24574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:02:54.986482 24573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.986456 24568 server_base.cc:1061] running on GCE node
W20260501 02:02:54.986362 24576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:02:54.987238 24568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:02:54.988092 24568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:02:54.989267 24568 hybrid_clock.cc:648] HybridClock initialized: now 1777600974989240 us; error 39 us; skew 500 ppm
I20260501 02:02:54.991086 24568 webserver.cc:492] Webserver started at http://127.18.96.254:34625/ using document root <none> and password file <none>
I20260501 02:02:54.991638 24568 fs_manager.cc:362] Metadata directory not provided
I20260501 02:02:54.991696 24568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:02:54.991932 24568 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:02:54.993557 24568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "803da29c7f2348ba94a1504ba7a0fd6a"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "d95d7bd594baf6ec8ce45401ac4d07d0"
server_key_iv: "4a35860670ae6ec24c900416944b1c42"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.994109 24568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "803da29c7f2348ba94a1504ba7a0fd6a"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "d95d7bd594baf6ec8ce45401ac4d07d0"
server_key_iv: "4a35860670ae6ec24c900416944b1c42"
server_key_version: "encryptionkey@0"
I20260501 02:02:54.997085 24568 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:02:54.999217 24582 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:02:55.000154 24568 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:02:55.000281 24568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "803da29c7f2348ba94a1504ba7a0fd6a"
format_stamp: "Formatted at 2026-05-01 02:02:54 on dist-test-slave-2j90"
server_key: "d95d7bd594baf6ec8ce45401ac4d07d0"
server_key_iv: "4a35860670ae6ec24c900416944b1c42"
server_key_version: "encryptionkey@0"
I20260501 02:02:55.000372 24568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:02:55.016484 24568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:02:55.017035 24568 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:02:55.017227 24568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:02:55.023810 24568 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44587
I20260501 02:02:55.023831 24634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44587 every 8 connection(s)
I20260501 02:02:55.024678 24568 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:02:55.027251 24635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:02:55.029354 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24568
I20260501 02:02:55.029510 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:02:55.029791 18819 external_mini_cluster.cc:1468] Setting key f37751ffbe90dcc6a6ce7e2b86672dfa
I20260501 02:02:55.032514 24635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a: Bootstrap starting.
I20260501 02:02:55.034394 24635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a: Neither blocks nor log segments found. Creating new log.
I20260501 02:02:55.035156 24635 log.cc:826] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:02:55.036621 24638 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:55.030854 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.18.96.254:44587)
0501 02:02:55.031341 (+ 487us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:55.031349 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:55.031377 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:02:55.031879 (+ 502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:55.031893 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:55.032179 (+ 286us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:55.032479 (+ 300us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:02:55.032944 (+ 465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.033628 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:55.034371 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.034640 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:55.035110 (+ 470us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:02:55.035126 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:02:55.035135 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:02:55.035566 (+ 431us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:02:55.035621 (+ 55us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:02:55.035638 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:02:55.035788 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:55.035954 (+ 166us) server_negotiation.cc:300] Negotiation successful
0501 02:02:55.036117 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260501 02:02:55.036986 24635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a: No bootstrap required, opened a new log
I20260501 02:02:55.039101 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:02:55.039494 24635 raft_consensus.cc:359] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } }
I20260501 02:02:55.039642 24635 raft_consensus.cc:385] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:02:55.039721 24635 raft_consensus.cc:740] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803da29c7f2348ba94a1504ba7a0fd6a, State: Initialized, Role: FOLLOWER
I20260501 02:02:55.040122 24635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } }
I20260501 02:02:55.040261 24635 raft_consensus.cc:399] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:02:55.040339 24635 raft_consensus.cc:493] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:02:55.040444 24635 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:02:55.041342 24635 raft_consensus.cc:515] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } }
I20260501 02:02:55.041682 24635 leader_election.cc:304] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 803da29c7f2348ba94a1504ba7a0fd6a; no voters:
I20260501 02:02:55.041994 24635 leader_election.cc:290] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:02:55.042081 24640 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:02:55.042317 24640 raft_consensus.cc:697] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [term 1 LEADER]: Becoming Leader. State: Replica: 803da29c7f2348ba94a1504ba7a0fd6a, State: Running, Role: LEADER
I20260501 02:02:55.042596 24640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } }
I20260501 02:02:55.043121 24635 sys_catalog.cc:565] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:02:55.044123 24641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } } }
I20260501 02:02:55.044255 24641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:55.044341 24642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 803da29c7f2348ba94a1504ba7a0fd6a. Latest consensus state: current_term: 1 leader_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "803da29c7f2348ba94a1504ba7a0fd6a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44587 } } }
I20260501 02:02:55.044433 24642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a [sys.catalog]: This master's current role is: LEADER
I20260501 02:02:55.045197 24649 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:02:55.047519 24649 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:02:55.051960 24649 catalog_manager.cc:1357] Generated new cluster ID: b75c5bd413cc4dd8b600f31e9a0dd6f3
I20260501 02:02:55.052047 24649 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:02:55.074278 24649 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:02:55.075039 24649 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:02:55.080605 24649 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 803da29c7f2348ba94a1504ba7a0fd6a: Generated new TSK 0
I20260501 02:02:55.081207 24649 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:02:55.120970 24638 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:55.091418 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.18.96.254:44587)
0501 02:02:55.091554 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:55.091559 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:55.091745 (+ 186us) server_negotiation.cc:408] Connection header received
0501 02:02:55.091925 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:55.091927 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:55.091972 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:55.092044 (+ 72us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:02:55.092492 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.092988 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:55.093895 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.094074 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:55.094138 (+ 64us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:02:55.120487 (+ 26349us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0501 02:02:55.120593 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:02:55.120774 (+ 181us) server_negotiation.cc:300] Negotiation successful
0501 02:02:55.120828 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51,"thread_start_us":134,"threads_started":1}
I20260501 02:02:55.136799 24638 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:02:55.127657 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49930 (local address 127.18.96.254:44587)
0501 02:02:55.127831 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0501 02:02:55.127835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:02:55.132014 (+ 4179us) server_negotiation.cc:408] Connection header received
0501 02:02:55.132753 (+ 739us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:02:55.132757 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:02:55.132801 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:02:55.132872 (+ 71us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:02:55.133591 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.134419 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:02:55.135301 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.135502 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:55.135562 (+ 60us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:02:55.136556 (+ 994us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 02:02:55.136714 (+ 158us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49930: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":62}
W20260501 02:02:55.136999 24638 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49930: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260501 02:02:55.138162 24671 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:02:55.127511 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44587 (local address 127.0.0.1:49930)
0501 02:02:55.131907 (+ 4396us) negotiation.cc:107] Waiting for socket to connect
0501 02:02:55.131923 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:02:55.132030 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:02:55.132984 (+ 954us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:02:55.132987 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:02:55.132996 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0501 02:02:55.133237 (+ 241us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:55.133246 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.134566 (+ 1320us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:02:55.134571 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:02:55.135179 (+ 608us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:02:55.135189 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:02:55.135299 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:02:55.135320 (+ 21us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0501 02:02:55.137145 (+ 1825us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0501 02:02:55.137346 (+ 201us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.254:44587: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":4314,"thread_start_us":93,"threads_started":1}
I20260501 02:03:10.147032 24677 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.143531 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.18.96.254:44587)
0501 02:03:10.143792 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.143797 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.143880 (+ 83us) server_negotiation.cc:408] Connection header received
0501 02:03:10.144074 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.144078 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.144127 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.144227 (+ 100us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:03:10.144612 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.145135 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.145833 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.146051 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.146095 (+ 44us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:03:10.146686 (+ 591us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 02:03:10.146865 (+ 179us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47986: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
W20260501 02:03:10.147078 24676 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:10.143350 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44587 (local address 127.0.0.1:47986)
0501 02:03:10.143771 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:10.143788 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:10.143949 (+ 161us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:10.144243 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:10.144247 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:10.144255 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0501 02:03:10.144489 (+ 234us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.144496 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.145256 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.145261 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:10.145714 (+ 453us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.145723 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.145845 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.145867 (+ 22us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0501 02:03:10.146831 (+ 964us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0501 02:03:10.146945 (+ 114us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.254:44587: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":167,"threads_started":1}
W20260501 02:03:10.147109 24677 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47986: JWT verification failed: token expired
W20260501 02:03:10.149502 24677 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:10.148729 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.18.96.254:44587)
0501 02:03:10.148861 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.148865 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.148983 (+ 118us) server_negotiation.cc:408] Connection header received
0501 02:03:10.149115 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.149118 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.149161 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.149252 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:10.149419 (+ 167us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47990: BlockingRecv error: recv got EOF from 127.0.0.1:47990 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260501 02:03:10.152834 24687 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.18.96.254:44587
W20260501 02:03:10.153349 24677 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:10.152513 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.18.96.254:44587)
0501 02:03:10.152662 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.152666 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.152776 (+ 110us) server_negotiation.cc:408] Connection header received
0501 02:03:10.152987 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.152991 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.153031 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.153115 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:10.153271 (+ 156us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47994: BlockingRecv error: recv got EOF from 127.0.0.1:47994 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260501 02:03:10.153591 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24568
2026-05-01T02:03:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260501 02:03:10.193451 18819 mini_oidc.cc:150] Starting JWKS server
I20260501 02:03:10.194391 18819 webserver.cc:492] Webserver started at https://127.0.0.1:35393/ using document root <none> and password file <none>
I20260501 02:03:10.194514 18819 mini_oidc.cc:186] Starting OIDC Discovery server
I20260501 02:03:10.194762 18819 webserver.cc:492] Webserver started at http://127.0.0.1:42769/ using document root <none> and password file <none>
2026-05-01T02:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:10Z Disabled control of system clock
I20260501 02:03:10.202277 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43931
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:43931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/testchainca.cert with env {}
W20260501 02:03:10.305570 24693 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:10.305784 24693 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:10.305833 24693 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:10.309125 24693 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:10.309183 24693 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:10.309203 24693 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:10.309226 24693 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:10.309291 24693 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260501 02:03:10.309332 24693 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260501 02:03:10.309398 24693 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:35393/jwks.json
I20260501 02:03:10.313710 24693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:43931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:10.314714 24693 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:10.315497 24693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:10.321372 24699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:10.321398 24698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.321448 24693 server_base.cc:1061] running on GCE node
W20260501 02:03:10.321386 24701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.322075 24693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:10.322928 24693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:10.324102 24693 hybrid_clock.cc:648] HybridClock initialized: now 1777600990324092 us; error 27 us; skew 500 ppm
I20260501 02:03:10.325753 24693 webserver.cc:492] Webserver started at http://127.18.96.254:34813/ using document root <none> and password file <none>
I20260501 02:03:10.326424 24693 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:10.326504 24693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:10.326722 24693 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:10.328260 24693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "a3f778b651b34c8b8bd72caf0a017a14"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "7eb2664e50c04d56d266589b3c873491"
server_key_iv: "fc143210783e2b6deb399f9771d0765a"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.328716 24693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "a3f778b651b34c8b8bd72caf0a017a14"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "7eb2664e50c04d56d266589b3c873491"
server_key_iv: "fc143210783e2b6deb399f9771d0765a"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.331672 24693 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:10.333664 24707 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:10.334583 24693 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:10.334712 24693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "a3f778b651b34c8b8bd72caf0a017a14"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "7eb2664e50c04d56d266589b3c873491"
server_key_iv: "fc143210783e2b6deb399f9771d0765a"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.334805 24693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:10.344760 24693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:10.345257 24693 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:10.345420 24693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:10.352777 24759 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:43931 every 8 connection(s)
I20260501 02:03:10.352785 24693 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:43931
I20260501 02:03:10.354045 24693 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:10.356776 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24693
I20260501 02:03:10.356894 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:10.357152 18819 external_mini_cluster.cc:1468] Setting key 54984c647aea677cf84c72b116ad1ebb
I20260501 02:03:10.357215 24760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:10.362749 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14: Bootstrap starting.
I20260501 02:03:10.364940 24760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:10.365177 24763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.358487 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.18.96.254:43931)
0501 02:03:10.359194 (+ 707us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.359201 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.359229 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:03:10.360290 (+ 1061us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.360306 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.360572 (+ 266us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.360840 (+ 268us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:10.361293 (+ 453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.361929 (+ 636us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.362641 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.362938 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.363382 (+ 444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:10.363452 (+ 70us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:10.363466 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:10.363936 (+ 470us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:10.363990 (+ 54us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:10.364024 (+ 34us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:10.364120 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:10.364311 (+ 191us) server_negotiation.cc:300] Negotiation successful
0501 02:03:10.364438 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":119,"threads_started":1}
I20260501 02:03:10.365742 24760 log.cc:826] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:10.367561 24760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14: No bootstrap required, opened a new log
I20260501 02:03:10.367892 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:03:10.370255 24760 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } }
I20260501 02:03:10.370445 24760 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:10.370519 24760 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f778b651b34c8b8bd72caf0a017a14, State: Initialized, Role: FOLLOWER
I20260501 02:03:10.370940 24760 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } }
I20260501 02:03:10.371084 24760 raft_consensus.cc:399] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:10.371169 24760 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:10.371261 24760 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:10.372030 24760 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } }
I20260501 02:03:10.372328 24760 leader_election.cc:304] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f778b651b34c8b8bd72caf0a017a14; no voters:
I20260501 02:03:10.372612 24760 leader_election.cc:290] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:10.372721 24765 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:10.372924 24765 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [term 1 LEADER]: Becoming Leader. State: Replica: a3f778b651b34c8b8bd72caf0a017a14, State: Running, Role: LEADER
I20260501 02:03:10.373216 24765 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } }
I20260501 02:03:10.373657 24760 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:10.374931 24767 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3f778b651b34c8b8bd72caf0a017a14. Latest consensus state: current_term: 1 leader_uuid: "a3f778b651b34c8b8bd72caf0a017a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } } }
I20260501 02:03:10.374948 24766 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3f778b651b34c8b8bd72caf0a017a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f778b651b34c8b8bd72caf0a017a14" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43931 } } }
I20260501 02:03:10.375058 24767 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.375058 24766 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.375978 24774 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:10.378747 24774 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:10.383689 24774 catalog_manager.cc:1357] Generated new cluster ID: 7b61e19ef87440b8abae7c257fd020c6
I20260501 02:03:10.383749 24774 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:10.402788 24774 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:10.403597 24774 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:10.413883 24774 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a3f778b651b34c8b8bd72caf0a017a14: Generated new TSK 0
I20260501 02:03:10.414449 24774 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260501 02:03:10.440357 24790 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260501 02:03:10.440428 24790 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260501 02:03:10.440894 24763 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.425425 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.18.96.254:43931)
0501 02:03:10.425571 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.425574 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.425743 (+ 169us) server_negotiation.cc:408] Connection header received
0501 02:03:10.425889 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.425891 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.425947 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.426090 (+ 143us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:03:10.426558 (+ 468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.427061 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.427938 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.428070 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.428145 (+ 75us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:03:10.440631 (+ 12486us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:03:10.440791 (+ 160us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260501 02:03:10.440982 24763 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35952: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260501 02:03:10.441303 24789 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:10.425335 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:43931 (local address 127.0.0.1:35952)
0501 02:03:10.425657 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:10.425672 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:10.425780 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:10.426122 (+ 342us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:10.426126 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:10.426135 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0501 02:03:10.426452 (+ 317us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.426460 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.427162 (+ 702us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.427173 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:10.427798 (+ 625us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.427807 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.427949 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.427970 (+ 21us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0501 02:03:10.440775 (+ 12805us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:03:10.440908 (+ 133us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.254:43931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
I20260501 02:03:10.441893 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24693
2026-05-01T02:03:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (287 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260501 02:03:10.481386 18819 mini_oidc.cc:150] Starting JWKS server
I20260501 02:03:10.482388 18819 webserver.cc:492] Webserver started at https://127.0.0.1:39873/ using document root <none> and password file <none>
I20260501 02:03:10.482524 18819 mini_oidc.cc:186] Starting OIDC Discovery server
I20260501 02:03:10.482764 18819 webserver.cc:492] Webserver started at http://127.0.0.1:36045/ using document root <none> and password file <none>
2026-05-01T02:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:10Z Disabled control of system clock
I20260501 02:03:10.490420 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43501
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:33765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:43501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260501 02:03:10.587719 24796 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:10.587909 24796 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:10.587951 24796 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:10.591143 24796 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:10.591204 24796 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:10.591221 24796 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:10.591265 24796 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:10.591303 24796 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260501 02:03:10.591336 24796 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260501 02:03:10.591369 24796 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:39873/jwks.json
I20260501 02:03:10.595093 24796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:43501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:10.596082 24796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:10.597093 24796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:10.602298 24802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:10.602334 24804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.602398 24796 server_base.cc:1061] running on GCE node
W20260501 02:03:10.602339 24801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.603025 24796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:10.603915 24796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:10.605108 24796 hybrid_clock.cc:648] HybridClock initialized: now 1777600990605078 us; error 42 us; skew 500 ppm
I20260501 02:03:10.606855 24796 webserver.cc:492] Webserver started at http://127.18.96.254:39909/ using document root <none> and password file <none>
I20260501 02:03:10.607411 24796 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:10.607513 24796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:10.607729 24796 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:10.609344 24796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "5833635b65a040138ad3a226f76bc758"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "cdc41019b53f8044833ae62226647d1c"
server_key_iv: "72b48092520718f5c87ab134d41b9a35"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.609813 24796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "5833635b65a040138ad3a226f76bc758"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "cdc41019b53f8044833ae62226647d1c"
server_key_iv: "72b48092520718f5c87ab134d41b9a35"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.612854 24796 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260501 02:03:10.615008 24810 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:10.615850 24796 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:10.615985 24796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "5833635b65a040138ad3a226f76bc758"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "cdc41019b53f8044833ae62226647d1c"
server_key_iv: "72b48092520718f5c87ab134d41b9a35"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.616097 24796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:10.654333 24796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:10.655061 24796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:10.655251 24796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:10.662529 24796 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:43501
I20260501 02:03:10.662544 24862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:43501 every 8 connection(s)
I20260501 02:03:10.663511 24796 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:10.665151 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24796
I20260501 02:03:10.665282 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:10.665535 18819 external_mini_cluster.cc:1468] Setting key e7ee3a339f15aa6ea910cc080c4e5736
I20260501 02:03:10.666360 24863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:10.671329 24863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758: Bootstrap starting.
I20260501 02:03:10.673269 24863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:10.674185 24863 log.cc:826] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:10.675678 24866 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.666690 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34918 (local address 127.18.96.254:43501)
0501 02:03:10.667199 (+ 509us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.667206 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.670283 (+ 3077us) server_negotiation.cc:408] Connection header received
0501 02:03:10.670846 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.670860 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.671156 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.671447 (+ 291us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:10.671937 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.672663 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.673317 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.673595 (+ 278us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.674194 (+ 599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:10.674211 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:10.674220 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:10.674652 (+ 432us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:10.674693 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:10.674712 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:10.674818 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:10.674995 (+ 177us) server_negotiation.cc:300] Negotiation successful
0501 02:03:10.675152 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260501 02:03:10.676029 24863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758: No bootstrap required, opened a new log
I20260501 02:03:10.678203 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:03:10.678962 24863 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } }
I20260501 02:03:10.679158 24863 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:10.679250 24863 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5833635b65a040138ad3a226f76bc758, State: Initialized, Role: FOLLOWER
I20260501 02:03:10.679683 24863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } }
I20260501 02:03:10.679790 24863 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:10.679875 24863 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:10.680006 24863 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:10.680776 24863 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } }
I20260501 02:03:10.681088 24863 leader_election.cc:304] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5833635b65a040138ad3a226f76bc758; no voters:
I20260501 02:03:10.681348 24863 leader_election.cc:290] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:10.681463 24868 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:10.681681 24868 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [term 1 LEADER]: Becoming Leader. State: Replica: 5833635b65a040138ad3a226f76bc758, State: Running, Role: LEADER
I20260501 02:03:10.682073 24868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } }
I20260501 02:03:10.682435 24863 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:10.683590 24870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5833635b65a040138ad3a226f76bc758. Latest consensus state: current_term: 1 leader_uuid: "5833635b65a040138ad3a226f76bc758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } } }
I20260501 02:03:10.683717 24870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.684100 24877 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:10.684537 24869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5833635b65a040138ad3a226f76bc758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5833635b65a040138ad3a226f76bc758" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43501 } } }
I20260501 02:03:10.684619 24869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.687044 24877 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:10.691742 24877 catalog_manager.cc:1357] Generated new cluster ID: 0040299757f54b448c947683a4f52527
I20260501 02:03:10.691802 24877 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:10.707720 24877 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:10.708454 24877 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:10.713511 24877 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5833635b65a040138ad3a226f76bc758: Generated new TSK 0
I20260501 02:03:10.714036 24877 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260501 02:03:10.743465 24893 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260501 02:03:10.743546 24893 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260501 02:03:10.744869 24866 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.726468 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.18.96.254:43501)
0501 02:03:10.726611 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.726615 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.726751 (+ 136us) server_negotiation.cc:408] Connection header received
0501 02:03:10.726864 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.726868 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.726922 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.727023 (+ 101us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:03:10.727352 (+ 329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.728687 (+ 1335us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.729310 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.729506 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.729582 (+ 76us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:03:10.744474 (+ 14892us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:03:10.744742 (+ 268us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260501 02:03:10.745009 24866 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260501 02:03:10.745113 24892 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:10.726365 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:43501 (local address 127.0.0.1:34932)
0501 02:03:10.726666 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:10.726686 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:10.726771 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:10.727030 (+ 259us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:10.727033 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:10.727039 (+ 6us) client_negotiation.cc:190] Negotiated authn=JWT
0501 02:03:10.727257 (+ 218us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.727262 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.728794 (+ 1532us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.728797 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:10.729208 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:10.729215 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.729299 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.729316 (+ 17us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0501 02:03:10.744621 (+ 15305us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0501 02:03:10.744876 (+ 255us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.254:43501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260501 02:03:10.746342 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24796
2026-05-01T02:03:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-05-01T02:03:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:10Z Disabled control of system clock
I20260501 02:03:10.793151 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38897
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38897
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/testchainca.cert with env {}
W20260501 02:03:10.889222 24897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:10.889390 24897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:10.889436 24897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:10.892645 24897 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:10.892695 24897 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:10.892712 24897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:10.892730 24897 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:10.892747 24897 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260501 02:03:10.892765 24897 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260501 02:03:10.892817 24897 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260501 02:03:10.896970 24897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:10.897825 24897 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:10.898550 24897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:10.903548 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:10.903529 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.903584 24897 server_base.cc:1061] running on GCE node
W20260501 02:03:10.903525 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:10.904243 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:10.905001 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:10.906168 24897 hybrid_clock.cc:648] HybridClock initialized: now 1777600990906150 us; error 41 us; skew 500 ppm
I20260501 02:03:10.907793 24897 webserver.cc:492] Webserver started at http://127.18.96.254:36973/ using document root <none> and password file <none>
I20260501 02:03:10.908330 24897 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:10.908409 24897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:10.908609 24897 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:10.910140 24897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "81867a00a619497c8e2ea326487327ae"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "5504f4d877583f7ac264086a312593f4"
server_key_iv: "0435008d6700fe8ef60681b9db272a58"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.910568 24897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "81867a00a619497c8e2ea326487327ae"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "5504f4d877583f7ac264086a312593f4"
server_key_iv: "0435008d6700fe8ef60681b9db272a58"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.913524 24897 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:10.915557 24911 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:10.916435 24897 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:03:10.916567 24897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "81867a00a619497c8e2ea326487327ae"
format_stamp: "Formatted at 2026-05-01 02:03:10 on dist-test-slave-2j90"
server_key: "5504f4d877583f7ac264086a312593f4"
server_key_iv: "0435008d6700fe8ef60681b9db272a58"
server_key_version: "encryptionkey@0"
I20260501 02:03:10.916662 24897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:10.928233 24897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:10.928766 24897 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:10.928952 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:10.935395 24963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38897 every 8 connection(s)
I20260501 02:03:10.935406 24897 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38897
I20260501 02:03:10.936328 24897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:10.937511 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 24897
I20260501 02:03:10.937628 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:10.937871 18819 external_mini_cluster.cc:1468] Setting key 7f2edef25d721550e84e22401b0fb9de
I20260501 02:03:10.939308 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:10.944293 24964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae: Bootstrap starting.
I20260501 02:03:10.946250 24964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:10.947008 24964 log.cc:826] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:10.948414 24967 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:10.938925 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.18.96.254:38897)
0501 02:03:10.939454 (+ 529us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:10.939462 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:10.939492 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:03:10.940074 (+ 582us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:10.940088 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:10.940369 (+ 281us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:10.940620 (+ 251us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:10.941172 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.942326 (+ 1154us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:10.943179 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:10.943355 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:10.946996 (+ 3641us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:10.947013 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:10.947022 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:10.947436 (+ 414us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:10.947476 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:10.947495 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:10.947596 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:10.947756 (+ 160us) server_negotiation.cc:300] Negotiation successful
0501 02:03:10.947913 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":127,"threads_started":1}
I20260501 02:03:10.948716 24964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae: No bootstrap required, opened a new log
I20260501 02:03:10.950734 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:03:10.951155 24964 raft_consensus.cc:359] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } }
I20260501 02:03:10.951359 24964 raft_consensus.cc:385] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:10.951426 24964 raft_consensus.cc:740] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81867a00a619497c8e2ea326487327ae, State: Initialized, Role: FOLLOWER
I20260501 02:03:10.951860 24964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } }
I20260501 02:03:10.952001 24964 raft_consensus.cc:399] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:10.952078 24964 raft_consensus.cc:493] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:10.952183 24964 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:10.952976 24964 raft_consensus.cc:515] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } }
I20260501 02:03:10.953269 24964 leader_election.cc:304] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81867a00a619497c8e2ea326487327ae; no voters:
I20260501 02:03:10.953517 24964 leader_election.cc:290] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:10.953627 24969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:10.953850 24969 raft_consensus.cc:697] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [term 1 LEADER]: Becoming Leader. State: Replica: 81867a00a619497c8e2ea326487327ae, State: Running, Role: LEADER
I20260501 02:03:10.954172 24969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } }
I20260501 02:03:10.954491 24964 sys_catalog.cc:565] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:10.955839 24970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81867a00a619497c8e2ea326487327ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } } }
I20260501 02:03:10.956118 24970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.956303 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81867a00a619497c8e2ea326487327ae. Latest consensus state: current_term: 1 leader_uuid: "81867a00a619497c8e2ea326487327ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81867a00a619497c8e2ea326487327ae" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38897 } } }
I20260501 02:03:10.956871 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:10.956928 24978 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:10.959327 24978 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:10.963613 24978 catalog_manager.cc:1357] Generated new cluster ID: a9408a5780c44bacb12cc64281b10e0f
I20260501 02:03:10.963672 24978 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:10.976164 24978 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:10.976882 24978 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:10.994576 24978 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 81867a00a619497c8e2ea326487327ae: Generated new TSK 0
I20260501 02:03:10.995211 24978 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:03:11.061698 24967 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:11.009468 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.18.96.254:38897)
0501 02:03:11.009632 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:11.009637 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:11.009811 (+ 174us) server_negotiation.cc:408] Connection header received
0501 02:03:11.010028 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:11.010031 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:11.010079 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:11.010205 (+ 126us) server_negotiation.cc:227] Negotiated authn=JWT
0501 02:03:11.010565 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:11.011086 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:11.011706 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:11.011886 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:11.011954 (+ 68us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0501 02:03:11.061256 (+ 49302us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0501 02:03:11.061542 (+ 286us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40924: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260501 02:03:11.061791 24967 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40924: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260501 02:03:11.061818 24993 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:11.009378 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38897 (local address 127.0.0.1:40924)
0501 02:03:11.009731 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:11.009742 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:11.009861 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:11.010206 (+ 345us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:11.010210 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:11.010218 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0501 02:03:11.010458 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:11.010464 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:11.011205 (+ 741us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:11.011211 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:11.011598 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:11.011605 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:11.011723 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:11.011744 (+ 21us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0501 02:03:11.061476 (+ 49732us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0501 02:03:11.061632 (+ 156us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.96.254:38897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260501 02:03:11.062489 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 24897
2026-05-01T02:03:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (295 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (110 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (107 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:11 dist-test-slave-2j90 krb5kdc[25005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:11 dist-test-slave-2j90 krb5kdc[25005](info): set up 2 sockets
May 01 02:03:11 dist-test-slave-2j90 krb5kdc[25005](info): commencing operation
krb5kdc: starting...
W20260501 02:03:13.242807 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.943s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:13.375260 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44409
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:13.470645 25021 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:13.470811 25021 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:13.470875 25021 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:13.473945 25021 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:13.474010 25021 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:13.474033 25021 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:13.474050 25021 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:13.474112 25021 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:13.479036 25021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:13.480186 25021 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:13.481159 25021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:13.486781 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:13.486811 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:13.486866 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:13.487157 25021 server_base.cc:1061] running on GCE node
I20260501 02:03:13.487630 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:13.488441 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:13.489609 25021 hybrid_clock.cc:648] HybridClock initialized: now 1777600993489581 us; error 45 us; skew 500 ppm
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:13.491968 25021 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:03:13.492900 25021 webserver.cc:492] Webserver started at http://127.18.96.254:40729/ using document root <none> and password file <none>
I20260501 02:03:13.493418 25021 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:13.493497 25021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:13.493690 25021 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:13.495318 25021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "951c54c2986642219fbb3729c408cd78"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "a15bd5030c3f35fc118601506611a478"
server_key_iv: "c986f77863f72cf35a6a81b7759a1e9b"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.495801 25021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "951c54c2986642219fbb3729c408cd78"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "a15bd5030c3f35fc118601506611a478"
server_key_iv: "c986f77863f72cf35a6a81b7759a1e9b"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.498834 25021 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:03:13.500777 25036 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:13.501622 25021 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:03:13.501752 25021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "951c54c2986642219fbb3729c408cd78"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "a15bd5030c3f35fc118601506611a478"
server_key_iv: "c986f77863f72cf35a6a81b7759a1e9b"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.501844 25021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:13.525051 25021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:13.527657 25021 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:13.527855 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:13.534538 25021 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44409
I20260501 02:03:13.534549 25088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44409 every 8 connection(s)
I20260501 02:03:13.535537 25021 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:13.538092 25089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:13.540480 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25021
I20260501 02:03:13.540616 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:13.540867 18819 external_mini_cluster.cc:1468] Setting key 8b71ff2926151fd63bac2b7a4c3b8e52
I20260501 02:03:13.543773 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78: Bootstrap starting.
I20260501 02:03:13.546012 25089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:13.546808 25089 log.cc:826] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78: Log is configured to *not* fsync() on all Append() calls
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:13.548677 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78: No bootstrap required, opened a new log
I20260501 02:03:13.550961 25089 raft_consensus.cc:359] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } }
I20260501 02:03:13.551119 25089 raft_consensus.cc:385] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:13.551187 25089 raft_consensus.cc:740] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951c54c2986642219fbb3729c408cd78, State: Initialized, Role: FOLLOWER
I20260501 02:03:13.551621 25089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } }
I20260501 02:03:13.551772 25089 raft_consensus.cc:399] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:13.551843 25089 raft_consensus.cc:493] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:13.551970 25089 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:13.552956 25089 raft_consensus.cc:515] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } }
I20260501 02:03:13.553284 25089 leader_election.cc:304] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 951c54c2986642219fbb3729c408cd78; no voters:
I20260501 02:03:13.553547 25089 leader_election.cc:290] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:13.553668 25094 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:13.553792 25092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:13.542179 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.18.96.254:44409)
0501 02:03:13.542561 (+ 382us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:13.542570 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:13.542600 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:03:13.543104 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:13.543118 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:13.543364 (+ 246us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:13.543603 (+ 239us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:13.544668 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.545536 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:13.546444 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.546701 (+ 257us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:13.549084 (+ 2383us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:13.549101 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:13.549111 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:13.549135 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:13.550847 (+ 1712us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:13.551369 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:13.551374 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:13.551379 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:13.551481 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:13.551777 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:13.551781 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:13.551785 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:13.552184 (+ 399us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:13.552319 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:13.552587 (+ 268us) server_negotiation.cc:300] Negotiation successful
0501 02:03:13.552805 (+ 218us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260501 02:03:13.553886 25094 raft_consensus.cc:697] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [term 1 LEADER]: Becoming Leader. State: Replica: 951c54c2986642219fbb3729c408cd78, State: Running, Role: LEADER
I20260501 02:03:13.554171 25094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } }
I20260501 02:03:13.554670 25089 sys_catalog.cc:565] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:13.555302 25096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 951c54c2986642219fbb3729c408cd78. Latest consensus state: current_term: 1 leader_uuid: "951c54c2986642219fbb3729c408cd78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } } }
I20260501 02:03:13.555414 25096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:13.556098 25095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "951c54c2986642219fbb3729c408cd78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "951c54c2986642219fbb3729c408cd78" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44409 } } }
I20260501 02:03:13.556203 25095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:13.556300 25101 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:13.559073 25101 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:13.563917 25101 catalog_manager.cc:1357] Generated new cluster ID: 3ac7ba1b08e44c13bca1ea782fb7e99f
I20260501 02:03:13.563985 25101 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:13.570936 25101 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:13.571911 25101 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:13.583334 25101 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 951c54c2986642219fbb3729c408cd78: Generated new TSK 0
I20260501 02:03:13.583874 25101 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:13.632791 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44409
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:13.730832 25117 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:13.731012 25117 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:13.731115 25117 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:13.734326 25117 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:13.734381 25117 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:13.734498 25117 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:13.738516 25117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:13.739450 25117 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:13.740280 25117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:13.746261 25125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:13.746272 25123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:13.746407 25117 server_base.cc:1061] running on GCE node
W20260501 02:03:13.746273 25122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:13.746862 25117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:13.747366 25117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:13.748515 25117 hybrid_clock.cc:648] HybridClock initialized: now 1777600993748485 us; error 42 us; skew 500 ppm
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:13.751145 25117 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:03:13.752138 25117 webserver.cc:492] Webserver started at http://127.18.96.193:41049/ using document root <none> and password file <none>
I20260501 02:03:13.752679 25117 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:13.752753 25117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:13.752952 25117 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:13.754634 25117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "30148bb7b0cb40d081fa1af1160172e2"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "3ada422c081d3bcd24542f29c8ee8502"
server_key_iv: "9a4d1938e283c75b2db4fe1fe8f3b0e3"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.755120 25117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "30148bb7b0cb40d081fa1af1160172e2"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "3ada422c081d3bcd24542f29c8ee8502"
server_key_iv: "9a4d1938e283c75b2db4fe1fe8f3b0e3"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.758301 25117 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:03:13.760340 25132 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:13.761220 25117 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:13.761339 25117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "30148bb7b0cb40d081fa1af1160172e2"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "3ada422c081d3bcd24542f29c8ee8502"
server_key_iv: "9a4d1938e283c75b2db4fe1fe8f3b0e3"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.761442 25117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:13.785688 25117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:13.788419 25117 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:13.788620 25117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:13.789112 25117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:13.789913 25117 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:13.790007 25117 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:13.790102 25117 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:13.790155 25117 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:13.799129 25117 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:43513
I20260501 02:03:13.799167 25245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:43513 every 8 connection(s)
I20260501 02:03:13.799966 25117 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:13.807649 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25117
I20260501 02:03:13.807797 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:13.808079 18819 external_mini_cluster.cc:1468] Setting key 10f06806223711e70e7e0503e2c4af28
I20260501 02:03:13.811329 25092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:13.801484 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57143 (local address 127.18.96.254:44409)
0501 02:03:13.801616 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:13.801619 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:13.802199 (+ 580us) server_negotiation.cc:408] Connection header received
0501 02:03:13.803008 (+ 809us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:13.803012 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:13.803081 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:13.803183 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:13.804583 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.805079 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:13.805718 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.805905 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:13.807870 (+ 1965us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:13.807892 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:13.807895 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:13.807926 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:13.809440 (+ 1514us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:13.809894 (+ 454us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:13.809900 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:13.809904 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:13.809989 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:13.810278 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:13.810283 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:13.810286 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:13.810476 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:13.810580 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:13.810959 (+ 379us) server_negotiation.cc:300] Negotiation successful
0501 02:03:13.811094 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:03:13.811877 25248 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:13.801696 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44409 (local address 127.18.96.193:57143)
0501 02:03:13.802076 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:13.802104 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:13.802792 (+ 688us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:13.803328 (+ 536us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:13.803336 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:13.803647 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:13.804423 (+ 776us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:13.804435 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.805198 (+ 763us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:13.805203 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:13.805601 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:13.805608 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:13.805809 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:13.806349 (+ 540us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:13.806365 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:13.807683 (+ 1318us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:13.809574 (+ 1891us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:13.809579 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:13.809586 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:13.809790 (+ 204us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:13.810102 (+ 312us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:13.810105 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:13.810106 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:13.810184 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:13.810577 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:13.810581 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:13.810741 (+ 160us) client_negotiation.cc:770] Sending connection context
0501 02:03:13.810881 (+ 140us) client_negotiation.cc:241] Negotiation successful
0501 02:03:13.811070 (+ 189us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
I20260501 02:03:13.813009 25246 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44409
I20260501 02:03:13.813262 25246 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:13.813791 25246 heartbeater.cc:507] Master 127.18.96.254:44409 requested a full tablet report, sending...
I20260501 02:03:13.815402 25053 ts_manager.cc:194] Registered new tserver with Master: 30148bb7b0cb40d081fa1af1160172e2 (127.18.96.193:43513)
I20260501 02:03:13.816996 25053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:57143
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:13.856381 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44409
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:13.952195 25253 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:13.952438 25253 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:13.952530 25253 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:13.955785 25253 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:13.955907 25253 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:13.956022 25253 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:13.960146 25253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:13.961032 25253 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:13.961769 25253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:13.967713 25258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:13.967741 25261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:13.967741 25259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:13.967824 25253 server_base.cc:1061] running on GCE node
I20260501 02:03:13.968209 25253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:13.968744 25253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:13.969914 25253 hybrid_clock.cc:648] HybridClock initialized: now 1777600993969893 us; error 38 us; skew 500 ppm
May 01 02:03:13 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:13.972383 25253 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:03:13.973308 25253 webserver.cc:492] Webserver started at http://127.18.96.194:39281/ using document root <none> and password file <none>
I20260501 02:03:13.973804 25253 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:13.973851 25253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:13.974097 25253 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:13.975719 25253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "8ae6e25162e4480cb06b4cc776554fe0"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "c6b36726e5b15c91e4c4d5b86494bb9f"
server_key_iv: "20caabab071fd065b7a85bccbe26feab"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.976179 25253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "8ae6e25162e4480cb06b4cc776554fe0"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "c6b36726e5b15c91e4c4d5b86494bb9f"
server_key_iv: "20caabab071fd065b7a85bccbe26feab"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.979238 25253 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260501 02:03:13.981253 25268 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:13.982290 25253 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:13.982373 25253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "8ae6e25162e4480cb06b4cc776554fe0"
format_stamp: "Formatted at 2026-05-01 02:03:13 on dist-test-slave-2j90"
server_key: "c6b36726e5b15c91e4c4d5b86494bb9f"
server_key_iv: "20caabab071fd065b7a85bccbe26feab"
server_key_version: "encryptionkey@0"
I20260501 02:03:13.982492 25253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:14.006363 25253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:14.008638 25253 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:14.008831 25253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:14.009359 25253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:14.010232 25253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:14.010277 25253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:14.010351 25253 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:14.010389 25253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:14.019181 25253 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:39695
I20260501 02:03:14.019222 25381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:39695 every 8 connection(s)
I20260501 02:03:14.020020 25253 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:14.021719 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25253
I20260501 02:03:14.021835 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:14.022157 18819 external_mini_cluster.cc:1468] Setting key ec994d0ccf9b76bbceeeff924ebe91b5
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600993, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:03:14.032724 25092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.021610 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55127 (local address 127.18.96.254:44409)
0501 02:03:14.021772 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.021777 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.022488 (+ 711us) server_negotiation.cc:408] Connection header received
0501 02:03:14.023345 (+ 857us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.023348 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.023403 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.023492 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.024897 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.025411 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.026260 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.026521 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.029204 (+ 2683us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:14.029221 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:14.029224 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:14.029255 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:14.031011 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.031521 (+ 510us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.031525 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.031527 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.031580 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.031841 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.031844 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.031845 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.031973 (+ 128us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:14.032049 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:14.032500 (+ 451us) server_negotiation.cc:300] Negotiation successful
0501 02:03:14.032592 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:03:14.033252 25384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.021892 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44409 (local address 127.18.96.194:55127)
0501 02:03:14.022349 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:14.022386 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:14.023155 (+ 769us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:14.023623 (+ 468us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:14.023630 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:14.023933 (+ 303us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:14.024735 (+ 802us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:14.024747 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.025535 (+ 788us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.025539 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:14.026125 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:14.026137 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.026322 (+ 185us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.026857 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:14.026883 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:14.028993 (+ 2110us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:14.031165 (+ 2172us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:14.031170 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:14.031178 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:14.031392 (+ 214us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:14.031684 (+ 292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:14.031686 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:14.031688 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:14.031761 (+ 73us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:14.032060 (+ 299us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:14.032076 (+ 16us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:14.032238 (+ 162us) client_negotiation.cc:770] Sending connection context
0501 02:03:14.032430 (+ 192us) client_negotiation.cc:241] Negotiation successful
0501 02:03:14.032589 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:03:14.034297 25382 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44409
I20260501 02:03:14.034556 25382 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:14.035113 25382 heartbeater.cc:507] Master 127.18.96.254:44409 requested a full tablet report, sending...
I20260501 02:03:14.036259 25053 ts_manager.cc:194] Registered new tserver with Master: 8ae6e25162e4480cb06b4cc776554fe0 (127.18.96.194:39695)
I20260501 02:03:14.036814 25053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:55127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:14.070080 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44409
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:14.167574 25389 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:14.167749 25389 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:14.167797 25389 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:14.170820 25389 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:14.170868 25389 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:14.170934 25389 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:14.174885 25389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:14.175757 25389 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:14.176427 25389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:14.182192 25394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:14.182260 25395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:14.182204 25397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:14.182585 25389 server_base.cc:1061] running on GCE node
I20260501 02:03:14.182926 25389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:14.183416 25389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:14.184559 25389 hybrid_clock.cc:648] HybridClock initialized: now 1777600994184537 us; error 34 us; skew 500 ppm
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.187101 25389 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:03:14.188043 25389 webserver.cc:492] Webserver started at http://127.18.96.195:35517/ using document root <none> and password file <none>
I20260501 02:03:14.188529 25389 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:14.188575 25389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:14.188763 25389 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:14.190361 25389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "725b80a6803c49889277048c64031d25"
format_stamp: "Formatted at 2026-05-01 02:03:14 on dist-test-slave-2j90"
server_key: "a0d56b015c951ef2fcd6621bfdd3fb14"
server_key_iv: "9ab5e7c4e0200aa02f81a2fe2da5582f"
server_key_version: "encryptionkey@0"
I20260501 02:03:14.190809 25389 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "725b80a6803c49889277048c64031d25"
format_stamp: "Formatted at 2026-05-01 02:03:14 on dist-test-slave-2j90"
server_key: "a0d56b015c951ef2fcd6621bfdd3fb14"
server_key_iv: "9ab5e7c4e0200aa02f81a2fe2da5582f"
server_key_version: "encryptionkey@0"
I20260501 02:03:14.193778 25389 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:14.195755 25404 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:14.196645 25389 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:14.196777 25389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "725b80a6803c49889277048c64031d25"
format_stamp: "Formatted at 2026-05-01 02:03:14 on dist-test-slave-2j90"
server_key: "a0d56b015c951ef2fcd6621bfdd3fb14"
server_key_iv: "9ab5e7c4e0200aa02f81a2fe2da5582f"
server_key_version: "encryptionkey@0"
I20260501 02:03:14.196869 25389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:14.208596 25389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:14.210816 25389 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:14.210994 25389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:14.211506 25389 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:14.212285 25389 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:14.212332 25389 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:14.212397 25389 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:14.212435 25389 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:14.221148 25389 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:42471
I20260501 02:03:14.221222 25517 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:42471 every 8 connection(s)
I20260501 02:03:14.222020 25389 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:14.224442 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25389
I20260501 02:03:14.224555 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:14.224818 18819 external_mini_cluster.cc:1468] Setting key 8aff412b76bf34d8d6fc4831d7f9d13e
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:14.232872 25092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.223578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:42521 (local address 127.18.96.254:44409)
0501 02:03:14.223711 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.223715 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.224325 (+ 610us) server_negotiation.cc:408] Connection header received
0501 02:03:14.225291 (+ 966us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.225294 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.225348 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.225444 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.227202 (+ 1758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.227706 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.228284 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.228412 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.230327 (+ 1915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:14.230340 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:14.230342 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:14.230361 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:14.231479 (+ 1118us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.231885 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.231888 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.231889 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.231924 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.232141 (+ 217us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.232143 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.232144 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.232258 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:14.232335 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:14.232684 (+ 349us) server_negotiation.cc:300] Negotiation successful
0501 02:03:14.232763 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:03:14.233433 25520 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.223792 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44409 (local address 127.18.96.195:42521)
0501 02:03:14.224150 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:14.224179 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:14.225096 (+ 917us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:14.225597 (+ 501us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:14.225607 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:14.226078 (+ 471us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:14.227069 (+ 991us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:14.227083 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.227813 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.227816 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:14.228189 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:14.228194 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.228321 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.228818 (+ 497us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:14.228835 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:14.230196 (+ 1361us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:14.231578 (+ 1382us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:14.231582 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:14.231590 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:14.231790 (+ 200us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:14.231995 (+ 205us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:14.231997 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:14.231999 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:14.232070 (+ 71us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:14.232345 (+ 275us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:14.232348 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:14.232500 (+ 152us) client_negotiation.cc:770] Sending connection context
0501 02:03:14.232645 (+ 145us) client_negotiation.cc:241] Negotiation successful
0501 02:03:14.232787 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260501 02:03:14.234431 25518 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44409
I20260501 02:03:14.234642 25518 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:14.235049 25518 heartbeater.cc:507] Master 127.18.96.254:44409 requested a full tablet report, sending...
I20260501 02:03:14.235946 25053 ts_manager.cc:194] Registered new tserver with Master: 725b80a6803c49889277048c64031d25 (127.18.96.195:42471)
I20260501 02:03:14.236464 25053 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:42521
I20260501 02:03:14.238374 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/test-admin.keytab.
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.253473 18819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260501 02:03:14.256150 25525 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:14.254719 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.18.96.193:43513)
0501 02:03:14.255046 (+ 327us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.255052 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.255072 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:03:14.255122 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.255126 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.255289 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.255458 (+ 169us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.255979 (+ 521us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37710: BlockingRecv error: recv got EOF from 127.0.0.1:37710 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":143,"threads_started":1}
W20260501 02:03:14.256286 18819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc' of type FILE
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.257449 18819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.258354 18819 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:03:14.266626 25525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.259167 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.18.96.193:43513)
0501 02:03:14.259262 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.259265 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.259335 (+ 70us) server_negotiation.cc:408] Connection header received
0501 02:03:14.259423 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.259425 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.259464 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.259526 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.260212 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.260915 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.261509 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.261668 (+ 159us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.263593 (+ 1925us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:14.263629 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:14.263639 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:14.263660 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:14.265085 (+ 1425us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.265551 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.265556 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.265557 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.265600 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.265797 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.265799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.265800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.265948 (+ 148us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:14.266156 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:14.266331 (+ 175us) server_negotiation.cc:300] Negotiation successful
0501 02:03:14.266498 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260501 02:03:14.270033 25525 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:14.269082 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.18.96.193:43513)
0501 02:03:14.269210 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.269214 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.269304 (+ 90us) server_negotiation.cc:408] Connection header received
0501 02:03:14.269436 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.269440 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.269480 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.269569 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.269917 (+ 348us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37734: BlockingRecv error: recv got EOF from 127.0.0.1:37734 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260501 02:03:14.270226 18819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc' of type FILE
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.271261 18819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.272087 18819 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:03:14.279481 25525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.272814 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.18.96.193:43513)
0501 02:03:14.272936 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.272940 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.273043 (+ 103us) server_negotiation.cc:408] Connection header received
0501 02:03:14.273113 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.273115 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.273150 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.273213 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.273739 (+ 526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.274285 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.274867 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.274996 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.276869 (+ 1873us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:14.276888 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:14.276893 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:14.276923 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:14.278399 (+ 1476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.278723 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.278726 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.278727 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.278762 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.278938 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.278940 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.278941 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.279040 (+ 99us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:14.279139 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:14.279302 (+ 163us) server_negotiation.cc:300] Negotiation successful
0501 02:03:14.279386 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260501 02:03:14.281947 25525 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:14.281042 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.18.96.193:43513)
0501 02:03:14.281163 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.281168 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.281250 (+ 82us) server_negotiation.cc:408] Connection header received
0501 02:03:14.281385 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.281388 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.281426 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.281513 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.281842 (+ 329us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37748: BlockingRecv error: recv got EOF from 127.0.0.1:37748 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260501 02:03:14.282150 18819 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777600936179445-18819-0/krb5kdc/krb5cc' of type FILE
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.283124 18819 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:14.283934 18819 init.cc:295] Successfully reacquired a new kerberos TGT
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:03:14.291991 25525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:14.284650 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.18.96.193:43513)
0501 02:03:14.284763 (+ 113us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:14.284766 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:14.284940 (+ 174us) server_negotiation.cc:408] Connection header received
0501 02:03:14.285005 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:14.285007 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:14.285039 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:14.285102 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:14.285666 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.286440 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:14.287122 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:14.287265 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:14.289094 (+ 1829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:14.289119 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:14.289121 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:14.289141 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:14.290591 (+ 1450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.291015 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.291019 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.291020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.291066 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:14.291302 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:14.291317 (+ 15us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:14.291318 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:14.291433 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:14.291553 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:14.291765 (+ 212us) server_negotiation.cc:300] Negotiation successful
0501 02:03:14.291853 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:03:14.292892 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25117
I20260501 02:03:14.298820 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25253
I20260501 02:03:14.304450 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25389
I20260501 02:03:14.310055 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25021
2026-05-01T02:03:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3032 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25538](info): set up 2 sockets
May 01 02:03:14 dist-test-slave-2j90 krb5kdc[25538](info): commencing operation
krb5kdc: starting...
W20260501 02:03:16.298799 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.967s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:16 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:16.432124 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37935
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:16.527815 25554 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:16.528007 25554 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:16.528055 25554 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:16.531311 25554 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:16.531359 25554 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:16.531409 25554 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:16.531447 25554 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:16.531478 25554 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:16.535470 25554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:37935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:16.536361 25554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:16.537111 25554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:16.542279 25562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:16.542286 25559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:16.542295 25560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:16.542646 25554 server_base.cc:1061] running on GCE node
I20260501 02:03:16.543098 25554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:16.543888 25554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:16.545037 25554 hybrid_clock.cc:648] HybridClock initialized: now 1777600996545022 us; error 29 us; skew 500 ppm
May 01 02:03:16 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:16.547557 25554 init.cc:377] Logged in from keytab as oryx/127.18.96.254@KRBTEST.COM (short username oryx)
I20260501 02:03:16.548575 25554 webserver.cc:492] Webserver started at http://127.18.96.254:44015/ using document root <none> and password file <none>
I20260501 02:03:16.549077 25554 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:16.549146 25554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:16.549376 25554 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:16.550951 25554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "71f3c8b755f34d6a8e7779286445a80f"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "3a270120999654410db8aeefc87db1ec"
server_key_iv: "dd1236e1be016382a6daf1e8e8f7325b"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.551424 25554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "71f3c8b755f34d6a8e7779286445a80f"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "3a270120999654410db8aeefc87db1ec"
server_key_iv: "dd1236e1be016382a6daf1e8e8f7325b"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.554409 25554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:03:16.556316 25569 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:16.557189 25554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:16.557327 25554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "71f3c8b755f34d6a8e7779286445a80f"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "3a270120999654410db8aeefc87db1ec"
server_key_iv: "dd1236e1be016382a6daf1e8e8f7325b"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.557420 25554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:16.568054 25554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:16.572757 25554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:16.572950 25554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:16.579880 25554 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:37935
I20260501 02:03:16.579890 25621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:37935 every 8 connection(s)
I20260501 02:03:16.580788 25554 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:16.583324 25622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:16.586774 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25554
I20260501 02:03:16.586905 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:16.587182 18819 external_mini_cluster.cc:1468] Setting key 100d2b0ab3bc7e6b279284c5e2579bc6
I20260501 02:03:16.588901 25622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Bootstrap starting.
I20260501 02:03:16.591163 25622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:16.591884 25622 log.cc:826] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:16.593895 25622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: No bootstrap required, opened a new log
May 01 02:03:16 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:16.596645 25622 raft_consensus.cc:359] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } }
I20260501 02:03:16.596832 25622 raft_consensus.cc:385] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:16.596918 25622 raft_consensus.cc:740] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f3c8b755f34d6a8e7779286445a80f, State: Initialized, Role: FOLLOWER
I20260501 02:03:16.597344 25622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } }
I20260501 02:03:16.597535 25622 raft_consensus.cc:399] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:16.597636 25622 raft_consensus.cc:493] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:16.597751 25622 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:16.598788 25622 raft_consensus.cc:515] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } }
I20260501 02:03:16.599095 25622 leader_election.cc:304] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71f3c8b755f34d6a8e7779286445a80f; no voters:
I20260501 02:03:16.599387 25622 leader_election.cc:290] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:16.599452 25627 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:16.599632 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:16.588485 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.18.96.254:37935)
0501 02:03:16.588888 (+ 403us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:16.588897 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:16.588926 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:03:16.589459 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:16.589472 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:16.589708 (+ 236us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:16.589936 (+ 228us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:16.590975 (+ 1039us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.591774 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:16.592610 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.592838 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:16.595234 (+ 2396us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:16.595259 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:16.595269 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:16.595294 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:16.596968 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:16.597404 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:16.597408 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:16.597412 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:16.597478 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:16.597762 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:16.597765 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:16.597766 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:16.598136 (+ 370us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:16.598302 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:16.598593 (+ 291us) server_negotiation.cc:300] Negotiation successful
0501 02:03:16.598800 (+ 207us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260501 02:03:16.599640 25627 raft_consensus.cc:697] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 LEADER]: Becoming Leader. State: Replica: 71f3c8b755f34d6a8e7779286445a80f, State: Running, Role: LEADER
I20260501 02:03:16.600142 25627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } }
I20260501 02:03:16.600428 25622 sys_catalog.cc:565] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:16.601537 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } }
I20260501 02:03:16.601652 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:16.601859 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f3c8b755f34d6a8e7779286445a80f. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } }
I20260501 02:03:16.601925 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:16.602031 25636 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:16.604710 25636 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:16.609670 25636 catalog_manager.cc:1357] Generated new cluster ID: e45cd13ad7af4b85b473412450a56b5b
I20260501 02:03:16.609756 25636 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:16.616179 25636 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:16.616895 25636 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:16.629555 25636 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Generated new TSK 0
I20260501 02:03:16.630218 25636 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:16.685209 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37935
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:16.782635 25650 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:16.782817 25650 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:16.782902 25650 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:16.786052 25650 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:16.786101 25650 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:16.786173 25650 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:16.790290 25650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:16.791175 25650 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:16.791890 25650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:16.797734 25656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:16.797712 25658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:16.797717 25655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:16.798015 25650 server_base.cc:1061] running on GCE node
I20260501 02:03:16.798413 25650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:16.798846 25650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:16.799988 25650 hybrid_clock.cc:648] HybridClock initialized: now 1777600996799965 us; error 35 us; skew 500 ppm
May 01 02:03:16 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:16.802645 25650 init.cc:377] Logged in from keytab as oryx/127.18.96.193@KRBTEST.COM (short username oryx)
I20260501 02:03:16.803548 25650 webserver.cc:492] Webserver started at http://127.18.96.193:43491/ using document root <none> and password file <none>
I20260501 02:03:16.804042 25650 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:16.804090 25650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:16.804291 25650 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:16.805788 25650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "e7015c4d1c65476abc167fa1a9733c30"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "03a2fcffa776687241f3ec87db4bb9c4"
server_key_iv: "09d9de949f6c1ff43aae7d8fcff3f2df"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.806275 25650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "e7015c4d1c65476abc167fa1a9733c30"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "03a2fcffa776687241f3ec87db4bb9c4"
server_key_iv: "09d9de949f6c1ff43aae7d8fcff3f2df"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.809259 25650 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:03:16.811290 25665 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:16.812175 25650 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:16.812302 25650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "e7015c4d1c65476abc167fa1a9733c30"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "03a2fcffa776687241f3ec87db4bb9c4"
server_key_iv: "09d9de949f6c1ff43aae7d8fcff3f2df"
server_key_version: "encryptionkey@0"
I20260501 02:03:16.812394 25650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:16.822934 25650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:16.825578 25650 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:16.825773 25650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:16.826435 25650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:16.827232 25650 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:16.827302 25650 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:16.827360 25650 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:16.827410 25650 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:16.835844 25650 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:45295
I20260501 02:03:16.835911 25778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:45295 every 8 connection(s)
I20260501 02:03:16.836727 25650 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:16.840236 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25650
I20260501 02:03:16.840339 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:16.840634 18819 external_mini_cluster.cc:1468] Setting key 2988d6d58d5c42586bd9c6adf16193ee
May 01 02:03:16 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:16.848987 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:16.838372 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:32919 (local address 127.18.96.254:37935)
0501 02:03:16.838512 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:16.838516 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:16.839070 (+ 554us) server_negotiation.cc:408] Connection header received
0501 02:03:16.839926 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:16.839929 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:16.839988 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:16.840079 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:16.841465 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.842034 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:16.842684 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.842915 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:16.845310 (+ 2395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:16.845335 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:16.845340 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:16.845375 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:16.847130 (+ 1755us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:16.847625 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:16.847628 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:16.847630 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:16.847687 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:16.847956 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:16.847959 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:16.847961 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:16.848151 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:16.848259 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:16.848677 (+ 418us) server_negotiation.cc:300] Negotiation successful
0501 02:03:16.848799 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:03:16.849421 25781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:16.838596 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:32919)
0501 02:03:16.838948 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:16.838975 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:16.839735 (+ 760us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:16.840223 (+ 488us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:16.840231 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:16.840725 (+ 494us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:16.841273 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:16.841286 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.842188 (+ 902us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:16.842191 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:16.842565 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:16.842572 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:16.842709 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:16.843735 (+ 1026us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:16.843752 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:16.845125 (+ 1373us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:16.847245 (+ 2120us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:16.847250 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:16.847258 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:16.847504 (+ 246us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:16.847788 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:16.847790 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:16.847792 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:16.847865 (+ 73us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:16.848253 (+ 388us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:16.848257 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:16.848432 (+ 175us) client_negotiation.cc:770] Sending connection context
0501 02:03:16.848593 (+ 161us) client_negotiation.cc:241] Negotiation successful
0501 02:03:16.848756 (+ 163us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:03:16.850582 25779 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:16.850778 25779 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:16.851248 25779 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.194@KRBTEST.COM" created.
I20260501 02:03:16.852588 25586 ts_manager.cc:194] Registered new tserver with Master: e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
I20260501 02:03:16.853672 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:32919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:16.889092 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37935
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:16.987648 25786 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:16.987848 25786 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:16.987954 25786 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:16.991315 25786 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:16.991374 25786 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:16.991516 25786 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:16.995669 25786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:16.996806 25786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:16.997596 25786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.003703 25792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.003708 25791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.003710 25794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.003952 25786 server_base.cc:1061] running on GCE node
I20260501 02:03:17.004381 25786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.004891 25786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.006026 25786 hybrid_clock.cc:648] HybridClock initialized: now 1777600997006012 us; error 25 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.008649 25786 init.cc:377] Logged in from keytab as oryx/127.18.96.194@KRBTEST.COM (short username oryx)
I20260501 02:03:17.009560 25786 webserver.cc:492] Webserver started at http://127.18.96.194:38061/ using document root <none> and password file <none>
I20260501 02:03:17.010116 25786 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.010166 25786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.010319 25786 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:17.011814 25786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "b12a7cdcf85747b09cd0afc131f2ece0"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "d615b247efed5d913751caf7e8dd72f7"
server_key_iv: "fe6d25f4d347e226a5732dc5d982f32a"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.012207 25786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "b12a7cdcf85747b09cd0afc131f2ece0"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "d615b247efed5d913751caf7e8dd72f7"
server_key_iv: "fe6d25f4d347e226a5732dc5d982f32a"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.015461 25786 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260501 02:03:17.017501 25801 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.018525 25786 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:03:17.018615 25786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "b12a7cdcf85747b09cd0afc131f2ece0"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "d615b247efed5d913751caf7e8dd72f7"
server_key_iv: "fe6d25f4d347e226a5732dc5d982f32a"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.018687 25786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.038983 25786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.041791 25786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:17.041947 25786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.042567 25786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:17.043445 25786 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:17.043493 25786 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.043531 25786 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:17.043546 25786 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.052305 25786 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:34991
I20260501 02:03:17.052346 25914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:34991 every 8 connection(s)
I20260501 02:03:17.053248 25786 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:17.054432 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25786
I20260501 02:03:17.054585 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:17.054872 18819 external_mini_cluster.cc:1468] Setting key fc3f986dc5c777bb1d7be0ddc2f758dd
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.195@KRBTEST.COM" created.
I20260501 02:03:17.066301 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.055088 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:48989 (local address 127.18.96.254:37935)
0501 02:03:17.055262 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.055267 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.056183 (+ 916us) server_negotiation.cc:408] Connection header received
0501 02:03:17.057322 (+ 1139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.057325 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.057368 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.057441 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.058921 (+ 1480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.059427 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.060152 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.060323 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.062987 (+ 2664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.063007 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.063009 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.063045 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.064421 (+ 1376us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.064932 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.064936 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.064938 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.064988 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.065285 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.065289 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.065291 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.065428 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.065543 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.066018 (+ 475us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.066141 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:03:17.066938 25917 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.055384 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:48989)
0501 02:03:17.056030 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.056079 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.057099 (+ 1020us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.057609 (+ 510us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.057623 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.058275 (+ 652us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.058770 (+ 495us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.058782 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.059560 (+ 778us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.059566 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.060021 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.060028 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.060252 (+ 224us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.061446 (+ 1194us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.061463 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.062840 (+ 1377us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.064582 (+ 1742us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.064588 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.064599 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.064827 (+ 228us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.065092 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.065095 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.065097 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.065182 (+ 85us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.065542 (+ 360us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.065549 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.065749 (+ 200us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.065943 (+ 194us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.066301 (+ 358us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":132,"threads_started":1}
I20260501 02:03:17.068008 25915 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:17.068269 25915 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:17.068830 25915 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
I20260501 02:03:17.069911 25586 ts_manager.cc:194] Registered new tserver with Master: b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:17.070509 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:48989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:17.100666 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37935
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:17.200253 25922 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:17.200452 25922 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:17.200538 25922 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:17.204124 25922 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:17.204183 25922 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:17.204317 25922 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:17.208492 25922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:17.209529 25922 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:17.210404 25922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.216405 25930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.216426 25922 server_base.cc:1061] running on GCE node
W20260501 02:03:17.216558 25927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.216367 25928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.216984 25922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.217499 25922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.218648 25922 hybrid_clock.cc:648] HybridClock initialized: now 1777600997218626 us; error 35 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.221282 25922 init.cc:377] Logged in from keytab as oryx/127.18.96.195@KRBTEST.COM (short username oryx)
I20260501 02:03:17.222307 25922 webserver.cc:492] Webserver started at http://127.18.96.195:41979/ using document root <none> and password file <none>
I20260501 02:03:17.222800 25922 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.222862 25922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.223109 25922 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:17.224613 25922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "4962797c6d7f432bac3c64963011b6c1"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "4526cd47e05f1122e2e80ef6bc4bd1dc"
server_key_iv: "5779887b5d3858c0b76461efba7ef0e6"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.225003 25922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "4962797c6d7f432bac3c64963011b6c1"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "4526cd47e05f1122e2e80ef6bc4bd1dc"
server_key_iv: "5779887b5d3858c0b76461efba7ef0e6"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.227974 25922 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:03:17.230026 25937 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.231002 25922 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:17.231129 25922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "4962797c6d7f432bac3c64963011b6c1"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "4526cd47e05f1122e2e80ef6bc4bd1dc"
server_key_iv: "5779887b5d3858c0b76461efba7ef0e6"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.231225 25922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.249109 25922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.251878 25922 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:17.252077 25922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.252624 25922 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:17.253451 25922 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:17.253528 25922 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.253594 25922 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:17.253643 25922 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.262992 25922 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:38149
I20260501 02:03:17.263027 26050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:38149 every 8 connection(s)
I20260501 02:03:17.263892 25922 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:17.265684 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 25922
I20260501 02:03:17.265807 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:17.266132 18819 external_mini_cluster.cc:1468] Setting key 6f0ce76dca753b08c8c224dc9661fbf6
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:17.275233 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.265595 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:54127 (local address 127.18.96.254:37935)
0501 02:03:17.265780 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.265786 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.266449 (+ 663us) server_negotiation.cc:408] Connection header received
0501 02:03:17.267264 (+ 815us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.267267 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.267331 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.267405 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.268529 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.269005 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.269566 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.269681 (+ 115us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.272169 (+ 2488us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.272184 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.272186 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.272208 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.273583 (+ 1375us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.274058 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.274061 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.274062 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.274100 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.274400 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.274402 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.274403 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.274539 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.274622 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.275031 (+ 409us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.275115 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260501 02:03:17.275846 26053 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.265856 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:54127)
0501 02:03:17.266326 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.266362 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.267109 (+ 747us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.267558 (+ 449us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.267566 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.267963 (+ 397us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.268413 (+ 450us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.268424 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.269114 (+ 690us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.269116 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.269464 (+ 348us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.269469 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.269612 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.270674 (+ 1062us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.270691 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.272019 (+ 1328us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.273682 (+ 1663us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.273687 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.273694 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.273903 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.274204 (+ 301us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.274210 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.274212 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.274317 (+ 105us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.274633 (+ 316us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.274637 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.274809 (+ 172us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.274978 (+ 169us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.275136 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":117,"threads_started":1}
I20260501 02:03:17.276867 26051 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:17.277089 26051 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:17.277554 26051 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
I20260501 02:03:17.278581 25586 ts_manager.cc:194] Registered new tserver with Master: 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149)
I20260501 02:03:17.279270 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:54127
I20260501 02:03:17.279861 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:17.324849 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:35673
--webserver_interface=127.18.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:37935,127.18.96.253:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:17.423159 26058 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:17.423411 26058 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:17.423527 26058 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:17.426702 26058 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:17.426770 26058 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:17.426808 26058 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:17.426859 26058 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:17.426894 26058 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:17.430895 26058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:37935,127.18.96.253:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
W20260501 02:03:17.431095 26058 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.96.254:37935,127.18.96.253:35673'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260501 02:03:17.431934 26058 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:17.432726 26058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.437630 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.437692 26066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.437667 26063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.438050 26058 server_base.cc:1061] running on GCE node
I20260501 02:03:17.438452 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.439203 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.440392 26058 hybrid_clock.cc:648] HybridClock initialized: now 1777600997440375 us; error 40 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.443065 26058 init.cc:377] Logged in from keytab as oryx/127.18.96.253@KRBTEST.COM (short username oryx)
I20260501 02:03:17.443998 26058 webserver.cc:492] Webserver started at http://127.18.96.253:45961/ using document root <none> and password file <none>
I20260501 02:03:17.444458 26058 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.444502 26058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.444653 26058 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:17.446194 26058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/instance:
uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "992e4c1fab7f11daa0571a5012f04e7d"
server_key_iv: "e1280e03552d1c0fb32c294ef79da81b"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.446601 26058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal/instance:
uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "992e4c1fab7f11daa0571a5012f04e7d"
server_key_iv: "e1280e03552d1c0fb32c294ef79da81b"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.449438 26058 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:17.451395 26073 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.452229 26058 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:03:17.452327 26058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "992e4c1fab7f11daa0571a5012f04e7d"
server_key_iv: "e1280e03552d1c0fb32c294ef79da81b"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.452404 26058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.460569 26058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.463076 26058 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:17.463207 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.469619 26058 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.253:35673
I20260501 02:03:17.469632 26125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.253:35673 every 8 connection(s)
I20260501 02:03:17.470599 26058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
I20260501 02:03:17.473055 26126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:17.476483 26126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:03:17.480320 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26058
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.253@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
I20260501 02:03:17.488392 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.478016 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.18.96.254:37935)
0501 02:03:17.478163 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.478167 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.478695 (+ 528us) server_negotiation.cc:408] Connection header received
0501 02:03:17.479307 (+ 612us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.479310 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.479357 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.479449 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.480474 (+ 1025us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.481176 (+ 702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.482130 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.482327 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.485019 (+ 2692us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.485036 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.485039 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.485061 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.486428 (+ 1367us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.486922 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.486924 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.486926 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.486968 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.487327 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.487331 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.487334 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.487518 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.487601 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.488169 (+ 568us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.488276 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:03:17.489059 26128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.478189 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44582)
0501 02:03:17.478558 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.478587 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.479143 (+ 556us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.479597 (+ 454us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.479604 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.479921 (+ 317us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.480325 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.480334 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.481331 (+ 997us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.481336 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.481936 (+ 600us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.481942 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.482223 (+ 281us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.482837 (+ 614us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.482859 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.484611 (+ 1752us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.486568 (+ 1957us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.486573 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.486582 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.486799 (+ 217us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.487057 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.487063 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.487067 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.487192 (+ 125us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.487651 (+ 459us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.487660 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.487905 (+ 245us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.488130 (+ 225us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.488355 (+ 225us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260501 02:03:17.490396 26126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:03:17.490895 26130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.481107 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.18.96.253:35673)
0501 02:03:17.481452 (+ 345us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.481457 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.481475 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:03:17.481544 (+ 69us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.481549 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.481679 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.481817 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.482721 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.483708 (+ 987us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.484520 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.484666 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.486884 (+ 2218us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.486906 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.486918 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.486942 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.488851 (+ 1909us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.489345 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.489350 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.489352 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.489404 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.489663 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.489666 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.489667 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.489831 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.490143 (+ 312us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.490614 (+ 471us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.490730 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.253@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
I20260501 02:03:17.496693 26128 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.490839 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:55962)
0501 02:03:17.490940 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.490953 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.491033 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.491277 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.491280 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.491404 (+ 124us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.491531 (+ 127us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.491536 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.492164 (+ 628us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.492169 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.492562 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.492568 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.492680 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.493028 (+ 348us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.493039 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.494109 (+ 1070us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.495572 (+ 1463us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.495577 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.495578 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.495720 (+ 142us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.495925 (+ 205us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.495927 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.495929 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.495968 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.496261 (+ 293us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.496264 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.496327 (+ 63us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.496432 (+ 105us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.496541 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260501 02:03:17.496717 26130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.491015 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.18.96.253:35673)
0501 02:03:17.491125 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.491128 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.491143 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:17.491164 (+ 21us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.491166 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.491193 (+ 27us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.491270 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.491637 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.492069 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.492664 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.492860 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.494217 (+ 1357us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.494238 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.494241 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.494261 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.495445 (+ 1184us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.495802 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.495806 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.495808 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.495844 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.496048 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.496052 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.496054 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.496154 (+ 100us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.496266 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.496473 (+ 207us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.496595 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":39}
I20260501 02:03:17.499018 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Bootstrap starting.
I20260501 02:03:17.500953 26126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:17.501559 26126 log.cc:826] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:17.503214 26126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: No bootstrap required, opened a new log
I20260501 02:03:17.505391 26126 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } }
I20260501 02:03:17.505524 26126 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:17.505599 26126 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2ec88cb58f42ed9c288ef50abc1b42, State: Initialized, Role: FOLLOWER
I20260501 02:03:17.506065 26126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } }
I20260501 02:03:17.506486 26132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } } }
I20260501 02:03:17.506598 26132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:17.507040 26126 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: configured and running, proceeding with master startup.
W20260501 02:03:17.513077 26143 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:03:17.514885 26143 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:03:17.518501 26058 master_runner.cc:428] Detected that this master 6b2ec88cb58f42ed9c288ef50abc1b42 is joining an existing cluster
I20260501 02:03:17.518544 26058 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.96.253:35673
I20260501 02:03:17.528043 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.520392 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.18.96.254:37935)
0501 02:03:17.520971 (+ 579us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.520975 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.520986 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:17.521079 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.521083 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.521129 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.521245 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.522010 (+ 765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.522578 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.523389 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.523588 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.524441 (+ 853us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.524461 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.524464 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.524490 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.526396 (+ 1906us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.526876 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.526880 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.526882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.526932 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.527176 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.527180 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.527182 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.527372 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.527468 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.527786 (+ 318us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.527877 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
I20260501 02:03:17.528256 26130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.520723 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.18.96.253:35673)
0501 02:03:17.520857 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.520861 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.521245 (+ 384us) server_negotiation.cc:408] Connection header received
0501 02:03:17.521366 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.521370 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.521409 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.521501 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.522091 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.522753 (+ 662us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.523848 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.524034 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.525053 (+ 1019us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.525068 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.525071 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.525097 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.526709 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.527121 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.527125 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.527128 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.527177 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.527472 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.527478 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.527482 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.527631 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.527727 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.527982 (+ 255us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.528104 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":51}
I20260501 02:03:17.528487 26149 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.520275 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44584)
0501 02:03:17.520880 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.520896 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.520979 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.521300 (+ 321us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.521304 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.521537 (+ 233us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.521864 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.521871 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.522703 (+ 832us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.522706 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.523266 (+ 560us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.523275 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.523388 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.523879 (+ 491us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.523895 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.524305 (+ 410us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.526522 (+ 2217us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.526529 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.526531 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.526799 (+ 268us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.527020 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.527024 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.527026 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.527075 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.527536 (+ 461us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.527538 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.527619 (+ 81us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.528213 (+ 594us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.528331 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":127,"threads_started":1}
I20260501 02:03:17.528453 26150 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.520697 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:55964)
0501 02:03:17.521129 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.521142 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.521261 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.521505 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.521508 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.521696 (+ 188us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.521898 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.521903 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.522886 (+ 983us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.522889 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.523420 (+ 531us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.523427 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.524117 (+ 690us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.524564 (+ 447us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.524575 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.524927 (+ 352us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.526816 (+ 1889us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.526820 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.526822 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.527025 (+ 203us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.527281 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.527284 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.527287 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.527361 (+ 74us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.527724 (+ 363us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.527728 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.527803 (+ 75us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.527920 (+ 117us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.528039 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":106,"threads_started":1}
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.254@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.252@KRBTEST.COM" created.
I20260501 02:03:17.561834 26130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.532330 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.18.96.253:35673)
0501 02:03:17.532431 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.532434 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.534268 (+ 1834us) server_negotiation.cc:408] Connection header received
0501 02:03:17.534465 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.534468 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.534516 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.534608 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.535237 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.535656 (+ 419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.536193 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.536372 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.538107 (+ 1735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.538128 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.538130 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.538149 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.559851 (+ 21702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.560508 (+ 657us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.560512 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.560514 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.560566 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.560808 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.560811 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.560813 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.561042 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.561148 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.561559 (+ 411us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.561674 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260501 02:03:17.561838 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.532275 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:55978)
0501 02:03:17.534164 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.534190 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.534343 (+ 153us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.534641 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.534646 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.534951 (+ 305us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.535124 (+ 173us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.535129 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.535760 (+ 631us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.535764 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.536089 (+ 325us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.536094 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.536209 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.536687 (+ 478us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.536698 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.537991 (+ 1293us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.560071 (+ 22080us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.560083 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.560087 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.560399 (+ 312us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.560658 (+ 259us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.560660 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.560662 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.560708 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.561163 (+ 455us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.561168 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.561268 (+ 100us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.561382 (+ 114us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.561609 (+ 227us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"thread_start_us":97,"threads_started":1}
I20260501 02:03:17.563292 25585 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.18.96.253:35673: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
member_type: NON_VOTER
last_known_addr {
host: "127.18.96.253"
port: 35673
}
attrs {
promote: true
}
}
dest_uuid: "71f3c8b755f34d6a8e7779286445a80f"
cas_config_opid_index: -1
I20260501 02:03:17.563961 25585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } }
I20260501 02:03:17.565450 26155 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } }
I20260501 02:03:17.566275 26100 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:17.567459 26155 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.18.96.253:35673
I20260501 02:03:17.567759 26100 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:17.567807 26154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } }
I20260501 02:03:17.567902 26154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:17.568423 26154 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:17.568619 26058 master.cc:562] Master@127.18.96.253:35673 shutting down...
W20260501 02:03:17.570685 25571 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260501 02:03:17.571585 25571 consensus_peers.cc:597] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f -> Peer 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253:35673): Couldn't send request to peer 6b2ec88cb58f42ed9c288ef50abc1b42. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260501 02:03:17.573926 26058 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:17.574254 26058 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:17.574399 26058 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: stopping tablet replica
WARNING: no policy specified for HTTP/127.18.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.252@KRBTEST.COM" created.
I20260501 02:03:17.588644 26058 master.cc:584] Master@127.18.96.253:35673 shutdown complete.
I20260501 02:03:17.588692 26058 master_runner.cc:305] Clearing existing system tablet
I20260501 02:03:17.588896 26058 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.588979 26058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.590837 26058 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.591789 26161 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.592013 26058 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.592123 26058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.592154 26058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "992e4c1fab7f11daa0571a5012f04e7d"
server_key_iv: "e1280e03552d1c0fb32c294ef79da81b"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.594033 26058 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:17.595142 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:39139
--webserver_interface=127.18.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:03:17.595822 26058 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260501 02:03:17.595878 26058 log.cc:1199] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260501 02:03:17.596036 26058 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Deleting consensus metadata
I20260501 02:03:17.596436 26058 master_runner.cc:315] Copying system tablet from 127.18.96.254:37935
I20260501 02:03:17.597551 26058 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.96.254:37935
I20260501 02:03:17.604454 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.598349 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.18.96.254:37935)
0501 02:03:17.598483 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.598504 (+ 21us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.598519 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:17.598557 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.598559 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.598597 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.598684 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.599261 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.599743 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.600351 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.600568 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.601363 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.601385 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.601387 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.601427 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.602980 (+ 1553us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.603432 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.603436 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.603438 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.603493 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.603731 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.603735 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.603737 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.603873 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.603979 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.604220 (+ 241us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.604318 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:03:17.604516 26168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.598072 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44600)
0501 02:03:17.598306 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.598321 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.598423 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.598699 (+ 276us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.598702 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.598929 (+ 227us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.599144 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.599149 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.599845 (+ 696us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.599850 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.600228 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.600234 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.600365 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.600936 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.600975 (+ 39us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.601252 (+ 277us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.603128 (+ 1876us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.603131 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.603134 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.603329 (+ 195us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.603586 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.603590 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.603592 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.603643 (+ 51us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.603969 (+ 326us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.603973 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.604055 (+ 82us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.604158 (+ 103us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.604293 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260501 02:03:17.605374 25606 tablet_copy_service.cc:140] P 71f3c8b755f34d6a8e7779286445a80f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6b2ec88cb58f42ed9c288ef50abc1b42 ({username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:44600)
I20260501 02:03:17.605523 25606 tablet_copy_service.cc:161] P 71f3c8b755f34d6a8e7779286445a80f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6b2ec88cb58f42ed9c288ef50abc1b42 at {username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:44600: session id = 6b2ec88cb58f42ed9c288ef50abc1b42-00000000000000000000000000000000
I20260501 02:03:17.606726 25606 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Tablet Copy: opened 0 blocks and 1 log segments
I20260501 02:03:17.608111 26058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260501 02:03:17.610049 26058 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: tablet copy: Starting download of 0 data blocks...
I20260501 02:03:17.610198 26058 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: tablet copy: Starting download of 1 WAL segments...
I20260501 02:03:17.611910 26058 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260501 02:03:17.613528 25606 tablet_copy_service.cc:342] P 71f3c8b755f34d6a8e7779286445a80f: Request end of tablet copy session 6b2ec88cb58f42ed9c288ef50abc1b42-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:44600
I20260501 02:03:17.613636 25606 tablet_copy_service.cc:434] P 71f3c8b755f34d6a8e7779286445a80f: ending tablet copy session 6b2ec88cb58f42ed9c288ef50abc1b42-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6b2ec88cb58f42ed9c288ef50abc1b42
W20260501 02:03:17.618382 26058 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260501 02:03:17.620581 26058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.622320 26172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.622465 26058 server_base.cc:1061] running on GCE node
W20260501 02:03:17.622547 26170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.622372 26169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.622871 26058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.623134 26058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.624280 26058 hybrid_clock.cc:648] HybridClock initialized: now 1777600997624265 us; error 30 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.625849 26058 init.cc:377] Logged in from keytab as oryx/127.18.96.253@KRBTEST.COM (short username oryx)
I20260501 02:03:17.626298 26058 webserver.cc:492] Webserver started at http://127.18.96.253:37819/ using document root <none> and password file <none>
I20260501 02:03:17.626462 26058 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.626506 26058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.627658 26058 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.628628 26179 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.628922 26058 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.628993 26058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "992e4c1fab7f11daa0571a5012f04e7d"
server_key_iv: "e1280e03552d1c0fb32c294ef79da81b"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.629056 26058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.646209 26058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.648010 26058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.652350 26058 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.253:35673
I20260501 02:03:17.652540 26231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.253:35673 every 8 connection(s)
I20260501 02:03:17.653029 26058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
I20260501 02:03:17.654747 26232 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:03:17.655701 26232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Bootstrap starting.
I20260501 02:03:17.665098 26232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:03:17.665374 26232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Bootstrap complete.
I20260501 02:03:17.665905 26232 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } }
I20260501 02:03:17.666046 26232 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b2ec88cb58f42ed9c288ef50abc1b42, State: Initialized, Role: LEARNER
I20260501 02:03:17.666177 26232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } }
I20260501 02:03:17.666498 26235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } }
I20260501 02:03:17.666630 26232 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:17.666653 26235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: LEARNER
I20260501 02:03:17.669812 26246 catalog_manager.cc:1269] Loaded cluster ID: e45cd13ad7af4b85b473412450a56b5b
I20260501 02:03:17.669860 26246 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: loading cluster ID for follower catalog manager: success
I20260501 02:03:17.671733 26246 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: acquiring CA information for follower catalog manager: success
I20260501 02:03:17.672987 26246 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260501 02:03:17.708391 26162 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:17.708623 26162 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:17.708722 26162 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:17.712023 26162 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:17.712105 26162 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:17.712144 26162 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:17.712200 26162 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:17.712255 26162 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:17.716853 26162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:17.717744 26162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:17.718549 26162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.723661 26254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.723821 26162 server_base.cc:1061] running on GCE node
W20260501 02:03:17.723765 26251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.723663 26252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.724368 26162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.725142 26162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.726292 26162 hybrid_clock.cc:648] HybridClock initialized: now 1777600997726273 us; error 32 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.728879 26162 init.cc:377] Logged in from keytab as oryx/127.18.96.252@KRBTEST.COM (short username oryx)
I20260501 02:03:17.729743 26162 webserver.cc:492] Webserver started at http://127.18.96.252:45085/ using document root <none> and password file <none>
I20260501 02:03:17.730322 26162 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.730370 26162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.730561 26162 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:17.732052 26162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/instance:
uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "b1c929978590f53cefc3bb65d226269a"
server_key_iv: "b0d4ad85b5469870c43d9a5fa0a469e1"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.732438 26162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal/instance:
uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "b1c929978590f53cefc3bb65d226269a"
server_key_iv: "b0d4ad85b5469870c43d9a5fa0a469e1"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.735581 26162 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:17.737435 26261 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.738307 26162 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:17.738413 26162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "b1c929978590f53cefc3bb65d226269a"
server_key_iv: "b0d4ad85b5469870c43d9a5fa0a469e1"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.738497 26162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.751904 26162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.754326 26162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:17.754521 26162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.761334 26162 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.252:39139
I20260501 02:03:17.761348 26313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.252:39139 every 8 connection(s)
I20260501 02:03:17.762352 26162 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
I20260501 02:03:17.764950 26314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:17.768534 26314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:03:17.771382 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26162
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.252@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
I20260501 02:03:17.780817 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.770284 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.18.96.254:37935)
0501 02:03:17.770426 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.770430 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.770918 (+ 488us) server_negotiation.cc:408] Connection header received
0501 02:03:17.771674 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.771677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.771735 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.771840 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.773356 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.774055 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.774706 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.774835 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.777149 (+ 2314us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.777167 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.777170 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.777200 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.778981 (+ 1781us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.779432 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.779437 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.779440 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.779494 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.779768 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.779772 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.779774 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.779913 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.779986 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.780545 (+ 559us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.780656 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:03:17.781462 26316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.770393 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44604)
0501 02:03:17.770795 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.770824 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.771451 (+ 627us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.772030 (+ 579us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.772040 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.772579 (+ 539us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.773187 (+ 608us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.773205 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.774160 (+ 955us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.774163 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.774573 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.774579 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.774778 (+ 199us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.775359 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.775383 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.777024 (+ 1641us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.779097 (+ 2073us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.779104 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.779110 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.779327 (+ 217us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.779601 (+ 274us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.779603 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.779605 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.779677 (+ 72us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.780016 (+ 339us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.780024 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.780291 (+ 267us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.780480 (+ 189us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.780663 (+ 183us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":142,"threads_started":1}
I20260501 02:03:17.781461 26317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.772124 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.18.96.252:39139)
0501 02:03:17.772414 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.772418 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.772434 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:03:17.772487 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.772490 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.772619 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.772733 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.773670 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.774536 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.775289 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.775486 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.777599 (+ 2113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.777633 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.777643 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.777666 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.779680 (+ 2014us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.780093 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.780097 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.780098 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.780151 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.780379 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.780381 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.780383 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.780579 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.780739 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.780972 (+ 233us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.781101 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
I20260501 02:03:17.782528 26314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.252@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
I20260501 02:03:17.789069 26316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.782999 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:55982)
0501 02:03:17.783162 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.783176 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.783244 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.783581 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.783583 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.783706 (+ 123us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.783840 (+ 134us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.783845 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.784715 (+ 870us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.784717 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.785047 (+ 330us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.785052 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.785131 (+ 79us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.785531 (+ 400us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.785542 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.786653 (+ 1111us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.788159 (+ 1506us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.788162 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.788163 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.788312 (+ 149us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.788493 (+ 181us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.788495 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.788497 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.788533 (+ 36us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.788777 (+ 244us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.788778 (+ 1us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.788832 (+ 54us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.788900 (+ 68us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.788972 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260501 02:03:17.789131 26318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.783114 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.18.96.253:35673)
0501 02:03:17.783377 (+ 263us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.783381 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.783394 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:17.783437 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.783446 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.783488 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.783567 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.783940 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.784624 (+ 684us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.785135 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.785256 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.786762 (+ 1506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.786780 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.786782 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.786801 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.788070 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.788386 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.788389 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.788390 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.788424 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.788605 (+ 181us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.788607 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.788608 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.788710 (+ 102us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.788775 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.788946 (+ 171us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.789025 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260501 02:03:17.789844 26314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.252@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
I20260501 02:03:17.796217 26316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.790291 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:32982)
0501 02:03:17.790380 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.790408 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.790538 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.790780 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.790783 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.790921 (+ 138us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.791037 (+ 116us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.791041 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.791666 (+ 625us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.791674 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.791974 (+ 300us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.791979 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.792080 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.792389 (+ 309us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.792398 (+ 9us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.793478 (+ 1080us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.795063 (+ 1585us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.795068 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.795070 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.795224 (+ 154us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.795449 (+ 225us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.795454 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.795456 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.795500 (+ 44us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.795800 (+ 300us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.795803 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.795870 (+ 67us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.795963 (+ 93us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.796062 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260501 02:03:17.796231 26317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.790376 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.18.96.252:39139)
0501 02:03:17.790478 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.790483 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.790497 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:17.790625 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.790631 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.790676 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.790771 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.791145 (+ 374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.791562 (+ 417us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.792066 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.792262 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.793598 (+ 1336us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.793618 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.793620 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.793639 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.794940 (+ 1301us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.795318 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.795323 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.795325 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.795365 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.795590 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.795595 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.795597 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.795706 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.795800 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.796034 (+ 234us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.796110 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260501 02:03:17.798720 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Bootstrap starting.
I20260501 02:03:17.800818 26314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:17.801549 26314 log.cc:826] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:17.803174 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: No bootstrap required, opened a new log
I20260501 02:03:17.805473 26314 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } }
I20260501 02:03:17.805610 26314 raft_consensus.cc:385] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:17.805678 26314 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ee37aa6a7f422a96e94e8d5abd3bfc, State: Initialized, Role: FOLLOWER
I20260501 02:03:17.806099 26314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } }
I20260501 02:03:17.806478 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } } }
I20260501 02:03:17.806594 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:17.807005 26314 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: configured and running, proceeding with master startup.
W20260501 02:03:17.811172 26331 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:03:17.811266 26331 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:03:17.814333 26162 master_runner.cc:428] Detected that this master b2ee37aa6a7f422a96e94e8d5abd3bfc is joining an existing cluster
I20260501 02:03:17.814390 26162 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.96.252:39139
I20260501 02:03:17.838892 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.835593 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44616)
0501 02:03:17.835758 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.835772 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.835850 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.836243 (+ 393us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.836247 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.836279 (+ 32us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:17.836513 (+ 234us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.836519 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.837521 (+ 1002us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.837525 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.838476 (+ 951us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.838485 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.838603 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.838659 (+ 56us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.838711 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.838769 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260501 02:03:17.839334 26339 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.835673 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.18.96.254:37935)
0501 02:03:17.836037 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.836041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.836051 (+ 10us) server_negotiation.cc:408] Connection header received
0501 02:03:17.836090 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.836093 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.836129 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.836204 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:17.836640 (+ 436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.837403 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.838612 (+ 1209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.839058 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.839121 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.839167 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.839211 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":116,"threads_started":1}
I20260501 02:03:17.842917 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.840328 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:55990)
0501 02:03:17.840466 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.840478 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.840540 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.840763 (+ 223us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.840766 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.840781 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:17.840979 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.840984 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.841612 (+ 628us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.841616 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.842591 (+ 975us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.842600 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.842708 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.842720 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.842768 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.842813 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260501 02:03:17.843601 26318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.840426 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.18.96.253:35673)
0501 02:03:17.840549 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.840553 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.840565 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:03:17.840606 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.840608 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.840655 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.840739 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:17.841088 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.841518 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.842740 (+ 1222us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.843250 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.843370 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.843425 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.843483 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:03:17.849843 26337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.816587 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:32984)
0501 02:03:17.816889 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.816906 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.816999 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.817353 (+ 354us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.817357 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.843463 (+ 26106us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.843741 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.843749 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.844723 (+ 974us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.844729 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.845251 (+ 522us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.845269 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.845632 (+ 363us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.846122 (+ 490us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.846137 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.846523 (+ 386us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.848483 (+ 1960us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.848486 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.848488 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.848747 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.848954 (+ 207us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.848957 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.848959 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.849000 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.849348 (+ 348us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.849351 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.849417 (+ 66us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.849546 (+ 129us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.849654 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":130,"threads_started":1}
I20260501 02:03:17.850199 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.816573 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.18.96.254:37935)
0501 02:03:17.816724 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.816729 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.817365 (+ 636us) server_negotiation.cc:408] Connection header received
0501 02:03:17.817495 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.817498 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.817548 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.817627 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.842924 (+ 25297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.843621 (+ 697us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.844655 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.844838 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.845574 (+ 736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.845591 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.845597 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.845628 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.848374 (+ 2746us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.848913 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.848919 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.848921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.848964 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.849223 (+ 259us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.849227 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.849229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.849403 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.849496 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.849932 (+ 436us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.850066 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:03:17.850524 26317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.816962 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.18.96.252:39139)
0501 02:03:17.817057 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.817061 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.817074 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:17.817106 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.817110 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.817149 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.817224 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.843869 (+ 26645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.844532 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.845386 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.845548 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.846639 (+ 1091us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.846653 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.846656 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.846675 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.848375 (+ 1700us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.848836 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.848839 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.848841 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.848878 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.849072 (+ 194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.849075 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.849077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.849244 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.849345 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.850251 (+ 906us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.850360 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35,"spinlock_wait_cycles":5888}
I20260501 02:03:17.850698 26338 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.817023 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44612)
0501 02:03:17.817283 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.817294 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.817393 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.817640 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.817643 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.842179 (+ 24536us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.842791 (+ 612us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.842799 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.843763 (+ 964us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.843768 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.844314 (+ 546us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.844325 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.844426 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.844988 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.845006 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.845440 (+ 434us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.848483 (+ 3043us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.848488 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.848489 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.848806 (+ 317us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.849067 (+ 261us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.849071 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.849073 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.849134 (+ 61us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.849556 (+ 422us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.849559 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.849627 (+ 68us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.849721 (+ 94us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.849813 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"spinlock_wait_cycles":3968,"thread_start_us":113,"threads_started":1}
I20260501 02:03:17.856385 25779 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600996, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.254@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
I20260501 02:03:17.858414 26318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.845769 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.18.96.253:35673)
0501 02:03:17.846405 (+ 636us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.846410 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.847058 (+ 648us) server_negotiation.cc:408] Connection header received
0501 02:03:17.847205 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.847208 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.847248 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.847306 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.849776 (+ 2470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.850533 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.852828 (+ 2295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.853012 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.854443 (+ 1431us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.854463 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.854469 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.854495 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.856204 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.857302 (+ 1098us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.857306 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.857308 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.857352 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.857624 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.857628 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.857630 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.857772 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.857835 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.858159 (+ 324us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.858283 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521}
I20260501 02:03:17.859102 26340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.845921 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:56002)
0501 02:03:17.846990 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.847002 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.847111 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.847411 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.847413 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.849365 (+ 1952us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.849646 (+ 281us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.849652 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.851589 (+ 1937us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.851593 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.852499 (+ 906us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.852508 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.853473 (+ 965us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.853884 (+ 411us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.853899 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.854298 (+ 399us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.856965 (+ 2667us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.856969 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.856971 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.857157 (+ 186us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.857466 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.857469 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.857471 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.857532 (+ 61us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.857898 (+ 366us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.857901 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.857991 (+ 90us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.858833 (+ 842us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.858945 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":109,"threads_started":1}
I20260501 02:03:17.861776 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.854146 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33000)
0501 02:03:17.854299 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.854313 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.854396 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.854711 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.854715 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.854883 (+ 168us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.855061 (+ 178us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.855067 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.855912 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.855916 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.856433 (+ 517us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.856441 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.856555 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.856955 (+ 400us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.856967 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.858458 (+ 1491us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.860527 (+ 2069us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.860533 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.860535 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.860703 (+ 168us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.860951 (+ 248us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.860955 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.860957 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.861011 (+ 54us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.861340 (+ 329us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.861343 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.861421 (+ 78us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.861532 (+ 111us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.861640 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260501 02:03:17.861778 26317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.854198 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.18.96.252:39139)
0501 02:03:17.854294 (+ 96us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.854297 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.854375 (+ 78us) server_negotiation.cc:408] Connection header received
0501 02:03:17.854529 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.854532 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.854586 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.854680 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.855178 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.855790 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.856721 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.856892 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.858591 (+ 1699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.858625 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.858627 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.858651 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.860365 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.860803 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.860807 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.860809 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.860848 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.861096 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.861100 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.861102 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.861241 (+ 139us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.861345 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.861580 (+ 235us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.861663 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260501 02:03:17.862594 25586 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.18.96.252:39139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
member_type: NON_VOTER
last_known_addr {
host: "127.18.96.252"
port: 39139
}
attrs {
promote: true
}
}
dest_uuid: "71f3c8b755f34d6a8e7779286445a80f"
cas_config_opid_index: 5
I20260501 02:03:17.863109 25586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } }
I20260501 02:03:17.864081 26155 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.864478 26288 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:17.865228 26155 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.18.96.252:39139
I20260501 02:03:17.865394 26155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.865454 26155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:17.866076 26162 master.cc:562] Master@127.18.96.252:39139 shutting down...
I20260501 02:03:17.866292 26288 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260501 02:03:17.866755 26206 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 LEARNER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260501 02:03:17.866813 26154 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:17.867175 26155 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260501 02:03:17.868906 26206 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
W20260501 02:03:17.869161 25570 consensus_peers.cc:597] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f -> Peer b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139): Couldn't send request to peer b2ee37aa6a7f422a96e94e8d5abd3bfc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260501 02:03:17.870571 26235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f3c8b755f34d6a8e7779286445a80f. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.870658 26235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: LEARNER
I20260501 02:03:17.870707 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: NON_VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: true } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.870846 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: LEARNER
I20260501 02:03:17.871704 26162 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:17.871958 26162 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:17.872058 26162 tablet_replica.cc:333] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: stopping tablet replica
I20260501 02:03:17.872891 26154 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: attempting to promote NON_VOTER 6b2ec88cb58f42ed9c288ef50abc1b42 to VOTER
W20260501 02:03:17.872905 25586 master.cc:680] Remote error: unable to get registration information for peer b2ee37aa6a7f422a96e94e8d5abd3bfc: Service unavailable: service kudu.master.MasterService not registered on Master
I20260501 02:03:17.873293 26154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } }
I20260501 02:03:17.874439 26206 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 LEARNER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260501 02:03:17.874817 26343 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260501 02:03:17.876477 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.876166 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33014)
0501 02:03:17.876356 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.876417 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260501 02:03:17.876873 25570 consensus_peers.cc:597] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f -> Peer b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139): Couldn't send request to peer b2ee37aa6a7f422a96e94e8d5abd3bfc. Status: Network error: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 02:03:17.877115 26155 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.877569 26206 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.878249 26155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.878347 26155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:17.878494 26155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.878549 26155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:17.878831 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.879055 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:17.887629 26162 master.cc:584] Master@127.18.96.252:39139 shutdown complete.
I20260501 02:03:17.887693 26162 master_runner.cc:305] Clearing existing system tablet
I20260501 02:03:17.887938 26162 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.887981 26162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.889045 26162 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.889878 26345 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.890136 26162 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.890269 26162 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.890309 26162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "b1c929978590f53cefc3bb65d226269a"
server_key_iv: "b0d4ad85b5469870c43d9a5fa0a469e1"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.892231 26162 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:17.894434 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.894145 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33018)
0501 02:03:17.894308 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.894355 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260501 02:03:17.894673 25586 master.cc:680] Network error: unable to get registration information for peer b2ee37aa6a7f422a96e94e8d5abd3bfc: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111)
I20260501 02:03:17.894829 26162 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260501 02:03:17.894970 26162 log.cc:1199] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260501 02:03:17.895262 26162 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Deleting consensus metadata
I20260501 02:03:17.895462 26162 master_runner.cc:315] Copying system tablet from 127.18.96.254:37935
I20260501 02:03:17.896633 26162 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: tablet copy: Beginning tablet copy session from remote peer at address 127.18.96.254:37935
I20260501 02:03:17.903867 26351 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.897130 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44618)
0501 02:03:17.897443 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.897455 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.897548 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.897776 (+ 228us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.897778 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.897957 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:17.898246 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.898258 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.899264 (+ 1006us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.899267 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:17.899658 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.899664 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.899754 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.900343 (+ 589us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:17.900356 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:17.900627 (+ 271us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:17.902645 (+ 2018us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.902650 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.902655 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.902828 (+ 173us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.903065 (+ 237us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:17.903078 (+ 13us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:17.903079 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:17.903122 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:17.903493 (+ 371us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:17.903495 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:17.903567 (+ 72us) client_negotiation.cc:770] Sending connection context
0501 02:03:17.903649 (+ 82us) client_negotiation.cc:241] Negotiation successful
0501 02:03:17.903741 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260501 02:03:17.903976 25625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.897196 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.18.96.254:37935)
0501 02:03:17.897323 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.897327 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.897534 (+ 207us) server_negotiation.cc:408] Connection header received
0501 02:03:17.897631 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.897634 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.897685 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.897770 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:17.898357 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.899132 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:17.899761 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:17.899993 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:17.900737 (+ 744us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:17.900752 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:17.900755 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:17.900777 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:17.902505 (+ 1728us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.902922 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.902926 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.902928 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.902978 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:17.903208 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:17.903212 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:17.903214 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:17.903378 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:17.903479 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:17.903726 (+ 247us) server_negotiation.cc:300] Negotiation successful
0501 02:03:17.903830 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:03:17.904418 25606 tablet_copy_service.cc:140] P 71f3c8b755f34d6a8e7779286445a80f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b2ee37aa6a7f422a96e94e8d5abd3bfc ({username='oryx', principal='oryx/127.18.96.252@KRBTEST.COM'} at 127.0.0.1:44618)
I20260501 02:03:17.904533 25606 tablet_copy_service.cc:161] P 71f3c8b755f34d6a8e7779286445a80f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b2ee37aa6a7f422a96e94e8d5abd3bfc at {username='oryx', principal='oryx/127.18.96.252@KRBTEST.COM'} at 127.0.0.1:44618: session id = b2ee37aa6a7f422a96e94e8d5abd3bfc-00000000000000000000000000000000
I20260501 02:03:17.905304 25606 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Tablet Copy: opened 0 blocks and 1 log segments
I20260501 02:03:17.906484 26162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260501 02:03:17.908262 26162 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: tablet copy: Starting download of 0 data blocks...
I20260501 02:03:17.908476 26162 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: tablet copy: Starting download of 1 WAL segments...
I20260501 02:03:17.909893 26162 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260501 02:03:17.911276 25606 tablet_copy_service.cc:342] P 71f3c8b755f34d6a8e7779286445a80f: Request end of tablet copy session b2ee37aa6a7f422a96e94e8d5abd3bfc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.96.252@KRBTEST.COM'} at 127.0.0.1:44618
I20260501 02:03:17.911341 25606 tablet_copy_service.cc:434] P 71f3c8b755f34d6a8e7779286445a80f: ending tablet copy session b2ee37aa6a7f422a96e94e8d5abd3bfc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b2ee37aa6a7f422a96e94e8d5abd3bfc
W20260501 02:03:17.915990 26162 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260501 02:03:17.918329 26162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:17.919962 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:17.920038 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.920320 26162 server_base.cc:1061] running on GCE node
W20260501 02:03:17.920270 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:17.920626 26162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:17.920894 26162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:17.922024 26162 hybrid_clock.cc:648] HybridClock initialized: now 1777600997922010 us; error 27 us; skew 500 ppm
May 01 02:03:17 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600997, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:17.923694 26162 init.cc:377] Logged in from keytab as oryx/127.18.96.252@KRBTEST.COM (short username oryx)
I20260501 02:03:17.924157 26162 webserver.cc:492] Webserver started at http://127.18.96.252:45157/ using document root <none> and password file <none>
I20260501 02:03:17.924341 26162 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:17.924396 26162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:17.925405 26162 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.926272 26362 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:17.926522 26162 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:17.926577 26162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "b1c929978590f53cefc3bb65d226269a"
server_key_iv: "b0d4ad85b5469870c43d9a5fa0a469e1"
server_key_version: "encryptionkey@0"
I20260501 02:03:17.926682 26162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:17.930545 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.930252 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33020)
0501 02:03:17.930418 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.930461 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260501 02:03:17.930770 25586 master.cc:680] Network error: unable to get registration information for peer b2ee37aa6a7f422a96e94e8d5abd3bfc: Client connection negotiation failed: client connection to 127.18.96.252:39139: connect: Connection refused (error 111)
I20260501 02:03:17.935853 26162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:17.937364 26162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:17.941771 26162 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.252:39139
I20260501 02:03:17.941795 26414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.252:39139 every 8 connection(s)
I20260501 02:03:17.942291 26162 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
I20260501 02:03:17.943542 26415 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:03:17.944293 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Bootstrap starting.
I20260501 02:03:17.952178 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:03:17.952461 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: Bootstrap complete.
I20260501 02:03:17.952930 26415 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } }
I20260501 02:03:17.953027 26415 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2ee37aa6a7f422a96e94e8d5abd3bfc, State: Initialized, Role: LEARNER
I20260501 02:03:17.953136 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } }
I20260501 02:03:17.953450 26418 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:17.953545 26415 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:17.953580 26418 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: LEARNER
I20260501 02:03:17.957109 26429 catalog_manager.cc:1269] Loaded cluster ID: e45cd13ad7af4b85b473412450a56b5b
I20260501 02:03:17.957177 26429 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: loading cluster ID for follower catalog manager: success
I20260501 02:03:17.958712 26429 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: acquiring CA information for follower catalog manager: success
I20260501 02:03:17.959661 26429 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:03:18.002300 26151 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.998946 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33022)
0501 02:03:17.999104 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:17.999122 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:17.999205 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:17.999608 (+ 403us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:17.999611 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:17.999627 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:17.999941 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:17.999945 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.001190 (+ 1245us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.001193 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.001889 (+ 696us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.001896 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.002068 (+ 172us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.002082 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.002121 (+ 39us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.002170 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260501 02:03:18.002856 26430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:17.999051 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.18.96.252:39139)
0501 02:03:17.999343 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:17.999347 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:17.999361 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:17.999411 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:17.999414 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:17.999466 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:17.999573 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.000067 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.001071 (+ 1004us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.002063 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.002496 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.002613 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.002668 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.002719 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260501 02:03:18.072646 25915 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
I20260501 02:03:18.281101 26051 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
I20260501 02:03:18.340001 26389 raft_consensus.cc:1217] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260501 02:03:18.340528 26418 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f3c8b755f34d6a8e7779286445a80f. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:18.340693 26418 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: LEARNER
I20260501 02:03:18.882584 26431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: NON_VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: true } } }
I20260501 02:03:18.882761 26431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:18.883556 26431 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: attempting to promote NON_VOTER b2ee37aa6a7f422a96e94e8d5abd3bfc to VOTER
I20260501 02:03:18.883852 26431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:18.885051 26389 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 LEARNER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260501 02:03:18.885051 26206 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Refusing update from remote peer 71f3c8b755f34d6a8e7779286445a80f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260501 02:03:18.885439 26432 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260501 02:03:18.885732 26431 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260501 02:03:18.887668 26432 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.888114 26206 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.889398 26432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.889130 26389 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.889482 26432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:18.889617 26432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.889714 26432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:18.889925 26432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.890040 26432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:18.890022 26438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.890234 26438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:18.890990 26439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71f3c8b755f34d6a8e7779286445a80f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:18.891098 26439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:18.919060 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.912509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.18.96.253:35673)
0501 02:03:18.912719 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.912722 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.912818 (+ 96us) server_negotiation.cc:408] Connection header received
0501 02:03:18.912892 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.912894 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.912939 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.913019 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:18.913811 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.914457 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.915098 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.915230 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.916346 (+ 1116us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:18.916360 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:18.916362 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:18.916381 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:18.917820 (+ 1439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:18.918194 (+ 374us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:18.918197 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:18.918198 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:18.918234 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:18.918423 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:18.918424 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:18.918425 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:18.918530 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:18.918613 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.918827 (+ 214us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.918934 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260501 02:03:18.922278 26442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.919802 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44630)
0501 02:03:18.920040 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.920053 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.920149 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.920419 (+ 270us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.920422 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.920439 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.920611 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.920616 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.921325 (+ 709us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.921329 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.921878 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.921884 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.922024 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.922038 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.922107 (+ 69us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.922158 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260501 02:03:18.922803 26443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.919954 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.18.96.254:37935)
0501 02:03:18.920228 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.920233 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.920249 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:03:18.920282 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.920285 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.920321 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.920400 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.920714 (+ 314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.921206 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.922019 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.922602 (+ 583us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.922630 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.922670 (+ 40us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.922712 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260501 02:03:18.925926 26442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.923855 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:56024)
0501 02:03:18.924043 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.924053 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.924109 (+ 56us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.924372 (+ 263us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.924374 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.924389 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.924516 (+ 127us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.924519 (+ 3us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.925132 (+ 613us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.925134 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.925684 (+ 550us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.925689 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.925772 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.925782 (+ 10us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.925816 (+ 34us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.925850 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260501 02:03:18.926414 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.923948 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.18.96.253:35673)
0501 02:03:18.924070 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.924075 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.924088 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:18.924219 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.924223 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.924265 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.924357 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.924613 (+ 256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.925045 (+ 432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.925798 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.926228 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.926252 (+ 24us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.926289 (+ 37us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.926343 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260501 02:03:18.929612 26442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.927366 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33034)
0501 02:03:18.927483 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.927496 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.927572 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.927958 (+ 386us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.927960 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.927993 (+ 33us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.928115 (+ 122us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.928119 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.928782 (+ 663us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.928784 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.929366 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.929372 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.929455 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.929465 (+ 10us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.929499 (+ 34us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.929534 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260501 02:03:18.930263 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.927506 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.18.96.252:39139)
0501 02:03:18.927774 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.927778 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.927789 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:18.927828 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.927830 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.927869 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.927939 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.928215 (+ 276us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.928698 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.929475 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.929922 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.929948 (+ 26us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.930042 (+ 94us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.930167 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260501 02:03:18.937664 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.931799 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.18.96.252:39139)
0501 02:03:18.931905 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.931908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.931919 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:18.932019 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.932022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.932056 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.932118 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:18.932784 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.933220 (+ 436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.933760 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.933876 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.934980 (+ 1104us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:18.934993 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:18.934996 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:18.935016 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:18.936536 (+ 1520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:18.936882 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:18.936885 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:18.936886 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:18.936923 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:18.937098 (+ 175us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:18.937100 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:18.937101 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:18.937212 (+ 111us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:18.937321 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.937476 (+ 155us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.937562 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260501 02:03:18.940716 26445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.938429 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:44640)
0501 02:03:18.938701 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.938713 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.938790 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.938973 (+ 183us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.938976 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.938989 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.939144 (+ 155us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.939148 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.939890 (+ 742us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.939892 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.940437 (+ 545us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.940443 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.940538 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.940548 (+ 10us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.940589 (+ 41us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.940628 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260501 02:03:18.941111 26443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.938518 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.18.96.254:37935)
0501 02:03:18.938715 (+ 197us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.938718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.938756 (+ 38us) server_negotiation.cc:408] Connection header received
0501 02:03:18.938855 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.938857 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.938895 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.938957 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.939256 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.939798 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.940544 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.940918 (+ 374us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.940952 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.940991 (+ 39us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.941027 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260501 02:03:18.944744 26445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.942241 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.0.0.1:56030)
0501 02:03:18.942389 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.942401 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.942468 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.942678 (+ 210us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.942680 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.942699 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.942876 (+ 177us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.942880 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.943531 (+ 651us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.943535 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.944401 (+ 866us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.944410 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.944519 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.944531 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.944586 (+ 55us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.944629 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260501 02:03:18.945096 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.942332 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.18.96.253:35673)
0501 02:03:18.942435 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.942438 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.942449 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:18.942542 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.942544 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.942588 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.942648 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.942978 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.943428 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.944510 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.944901 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.944929 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.944974 (+ 45us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.945017 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260501 02:03:18.948638 26445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.946138 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.0.0.1:33052)
0501 02:03:18.946286 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:18.946297 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:18.946360 (+ 63us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:18.946557 (+ 197us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:18.946560 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:18.946576 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:18.946759 (+ 183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.946764 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.947439 (+ 675us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.947442 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:18.948288 (+ 846us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:18.948295 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.948397 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.948409 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:18.948453 (+ 44us) client_negotiation.cc:241] Negotiation successful
0501 02:03:18.948516 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260501 02:03:18.948951 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:18.946208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.18.96.252:39139)
0501 02:03:18.946305 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:18.946316 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:18.946334 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:03:18.946432 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:18.946434 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:18.946466 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:18.946526 (+ 60us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:18.946874 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.947352 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:18.948385 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:18.948740 (+ 355us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:18.948777 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:18.948831 (+ 54us) server_negotiation.cc:300] Negotiation successful
0501 02:03:18.948875 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260501 02:03:18.950109 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25650
I20260501 02:03:18.956632 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:45295
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:37935
--webserver_port=43491
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:19.054489 26446 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:19.054701 26446 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:19.054785 26446 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:19.058282 26446 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:19.058358 26446 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:19.058461 26446 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:19.062608 26446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:45295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=43491
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:19.063588 26446 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:19.064337 26446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:19.070559 26452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:19.070603 26454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:19.070595 26451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:19.070712 26446 server_base.cc:1061] running on GCE node
I20260501 02:03:19.071164 26446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:19.071636 26446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:19.072783 26446 hybrid_clock.cc:648] HybridClock initialized: now 1777600999072773 us; error 22 us; skew 500 ppm
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:19.075805 26446 init.cc:377] Logged in from keytab as oryx/127.18.96.193@KRBTEST.COM (short username oryx)
I20260501 02:03:19.076692 26446 webserver.cc:492] Webserver started at http://127.18.96.193:43491/ using document root <none> and password file <none>
I20260501 02:03:19.077185 26446 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:19.077265 26446 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:19.080472 26446 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:19.082191 26461 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.082998 26446 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:19.083109 26446 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "e7015c4d1c65476abc167fa1a9733c30"
format_stamp: "Formatted at 2026-05-01 02:03:16 on dist-test-slave-2j90"
server_key: "03a2fcffa776687241f3ec87db4bb9c4"
server_key_iv: "09d9de949f6c1ff43aae7d8fcff3f2df"
server_key_version: "encryptionkey@0"
I20260501 02:03:19.083501 26446 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:19.095731 26446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:19.097884 26446 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:19.098110 26446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:19.098611 26446 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:19.099481 26446 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:19.099548 26446 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.099606 26446 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:19.099643 26446 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.108022 26446 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:45295
I20260501 02:03:19.108047 26574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:45295 every 8 connection(s)
I20260501 02:03:19.108916 26446 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:19.111519 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26446
I20260501 02:03:19.111713 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25786
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
I20260501 02:03:19.120522 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:34991
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:37935
--webserver_port=38061
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:03:19.123932 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.110546 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57739 (local address 127.18.96.252:39139)
0501 02:03:19.110684 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.110688 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.114538 (+ 3850us) server_negotiation.cc:408] Connection header received
0501 02:03:19.114724 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.114730 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.114799 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.114847 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.115527 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.116254 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.117037 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.117238 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.119739 (+ 2501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.119762 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.119767 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.119794 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.121578 (+ 1784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122227 (+ 649us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.122232 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.122235 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.122292 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122873 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.122880 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.122883 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.123073 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.123186 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.123642 (+ 456us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.123761 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:03:19.123932 26443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.112818 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:53453 (local address 127.18.96.254:37935)
0501 02:03:19.112963 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.112969 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.112984 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:19.113026 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.113031 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.113084 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.113162 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.114812 (+ 1650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.115540 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.116381 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.116554 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.119548 (+ 2994us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.119568 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.119571 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.119594 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.121781 (+ 2187us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122346 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.122350 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.122354 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.122410 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122818 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.122824 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.122826 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.122972 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.123080 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.123646 (+ 566us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.123761 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:03:19.125080 26581 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.111679 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:53453)
0501 02:03:19.112536 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.112546 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.112639 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.113273 (+ 634us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.113279 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.114401 (+ 1122us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.114673 (+ 272us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.114683 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.115671 (+ 988us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.115674 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.116239 (+ 565us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.116245 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.116686 (+ 441us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.117517 (+ 831us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.117534 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.119427 (+ 1893us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.121920 (+ 2493us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.121939 (+ 19us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.121942 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.122219 (+ 277us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.122515 (+ 296us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122517 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.122518 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.122678 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.123080 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.123085 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.123336 (+ 251us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.124515 (+ 1179us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.124634 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"spinlock_wait_cycles":3072,"thread_start_us":90,"threads_started":1}
I20260501 02:03:19.125094 26580 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.110833 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.18.96.193:57739)
0501 02:03:19.111219 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.111268 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.114611 (+ 3343us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.114956 (+ 345us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.114959 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.115119 (+ 160us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.115379 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.115384 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.116378 (+ 994us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.116381 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.116915 (+ 534us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.116939 (+ 24us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.117062 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.117747 (+ 685us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.117767 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.119400 (+ 1633us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.121693 (+ 2293us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.121698 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.121707 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.122104 (+ 397us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.122707 (+ 603us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.122711 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.122714 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.122776 (+ 62us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.123169 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.123172 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.123336 (+ 164us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.124097 (+ 761us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.124360 (+ 263us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":21,"spinlock_wait_cycles":5376,"thread_start_us":115,"threads_started":1}
I20260501 02:03:19.125463 26579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.110833 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.18.96.193:48947)
0501 02:03:19.111218 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.111268 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.112244 (+ 976us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.112761 (+ 517us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.112773 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.113273 (+ 500us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.113862 (+ 589us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.113880 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.115022 (+ 1142us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.115028 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.115640 (+ 612us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.115652 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.116185 (+ 533us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.116763 (+ 578us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.116789 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.119595 (+ 2806us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.123544 (+ 3949us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.123551 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.123555 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.123779 (+ 224us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.124079 (+ 300us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.124085 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.124090 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.124147 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.124854 (+ 707us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.124860 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.124949 (+ 89us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.125065 (+ 116us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.125194 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"spinlock_wait_cycles":18688,"thread_start_us":90,"threads_started":1}
I20260501 02:03:19.125944 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.110529 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:48947 (local address 127.18.96.253:35673)
0501 02:03:19.110667 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.110672 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.111360 (+ 688us) server_negotiation.cc:408] Connection header received
0501 02:03:19.112432 (+ 1072us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.112439 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.112505 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.112598 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.114056 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.114874 (+ 818us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.115775 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.115999 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.119710 (+ 3711us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.119730 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.119733 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.119761 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.123441 (+ 3680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.123894 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.123911 (+ 17us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.123916 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.123983 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.124257 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.124263 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.124268 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.124482 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.124578 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.125602 (+ 1024us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.125733 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:03:19.126331 26576 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:19.126572 26576 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.126556 26577 heartbeater.cc:344] Connected to a master server at 127.18.96.253:35673
I20260501 02:03:19.126680 26577 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.127059 26577 heartbeater.cc:507] Master 127.18.96.253:35673 requested a full tablet report, sending...
I20260501 02:03:19.127060 26576 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
I20260501 02:03:19.128504 25586 ts_manager.cc:194] Re-registered known tserver with Master: e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
I20260501 02:03:19.128710 26196 ts_manager.cc:194] Registered new tserver with Master: e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
I20260501 02:03:19.129201 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:53453
I20260501 02:03:19.130769 26575 heartbeater.cc:344] Connected to a master server at 127.18.96.252:39139
I20260501 02:03:19.130875 26575 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.131136 26575 heartbeater.cc:507] Master 127.18.96.252:39139 requested a full tablet report, sending...
I20260501 02:03:19.132200 26379 ts_manager.cc:194] Registered new tserver with Master: e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
W20260501 02:03:19.221422 26582 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:19.221609 26582 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:19.221697 26582 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:19.224943 26582 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:19.224993 26582 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:19.225109 26582 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:19.229338 26582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:34991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=38061
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:19.230283 26582 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:19.230999 26582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:19.237407 26590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:19.237430 26588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:19.237644 26582 server_base.cc:1061] running on GCE node
W20260501 02:03:19.237430 26587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:19.238164 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:19.238605 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:19.239768 26582 hybrid_clock.cc:648] HybridClock initialized: now 1777600999239752 us; error 29 us; skew 500 ppm
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:19.242440 26582 init.cc:377] Logged in from keytab as oryx/127.18.96.194@KRBTEST.COM (short username oryx)
I20260501 02:03:19.243394 26582 webserver.cc:492] Webserver started at http://127.18.96.194:38061/ using document root <none> and password file <none>
I20260501 02:03:19.243878 26582 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:19.243934 26582 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:19.246984 26582 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:03:19.248628 26597 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.249518 26582 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:19.249650 26582 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "b12a7cdcf85747b09cd0afc131f2ece0"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "d615b247efed5d913751caf7e8dd72f7"
server_key_iv: "fe6d25f4d347e226a5732dc5d982f32a"
server_key_version: "encryptionkey@0"
I20260501 02:03:19.250053 26582 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:19.275463 26582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:19.278046 26582 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:19.278234 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:19.278776 26582 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:19.279644 26582 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:19.279713 26582 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.279778 26582 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:19.279825 26582 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.289921 26582 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:34991
I20260501 02:03:19.289939 26710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:34991 every 8 connection(s)
I20260501 02:03:19.290839 26582 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:19.297282 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26582
I20260501 02:03:19.297451 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25922
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
I20260501 02:03:19.305351 26443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.293852 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55393 (local address 127.18.96.254:37935)
0501 02:03:19.294061 (+ 209us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.294065 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.294187 (+ 122us) server_negotiation.cc:408] Connection header received
0501 02:03:19.294691 (+ 504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.294695 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.294766 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.294855 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.296093 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.296859 (+ 766us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.297612 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.297808 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.301186 (+ 3378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.301223 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.301227 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.301251 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.303199 (+ 1948us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.303828 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.303834 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.303836 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.303883 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304212 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.304216 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.304217 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.304391 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.304461 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.305066 (+ 605us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.305176 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260501 02:03:19.305361 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.292470 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:53483 (local address 127.18.96.252:39139)
0501 02:03:19.292632 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.292636 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.293425 (+ 789us) server_negotiation.cc:408] Connection header received
0501 02:03:19.294724 (+ 1299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.294727 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.294793 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.294845 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.296308 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.296788 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.298220 (+ 1432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.298877 (+ 657us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.301186 (+ 2309us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.301204 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.301209 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.301251 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.303743 (+ 2492us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304166 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.304172 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.304176 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.304223 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304496 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.304500 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.304502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.304658 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.304729 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.305066 (+ 337us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.305176 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:03:19.305464 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.292642 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46759 (local address 127.18.96.253:35673)
0501 02:03:19.292753 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.292756 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.293425 (+ 669us) server_negotiation.cc:408] Connection header received
0501 02:03:19.294724 (+ 1299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.294728 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.294793 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.294845 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.296201 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.296895 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.297738 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.297976 (+ 238us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.300998 (+ 3022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.301021 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.301027 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.301061 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.303476 (+ 2415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304005 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.304012 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.304016 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.304067 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304340 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.304346 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.304351 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.304530 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.304605 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.305077 (+ 472us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.305202 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260501 02:03:19.306720 26716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.292832 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.18.96.194:53483)
0501 02:03:19.293290 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.293333 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.294505 (+ 1172us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.295002 (+ 497us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.295009 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.295954 (+ 945us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.296192 (+ 238us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.296204 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.296904 (+ 700us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.296907 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.297475 (+ 568us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.297482 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.298987 (+ 1505us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.299439 (+ 452us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.299454 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.301024 (+ 1570us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.303856 (+ 2832us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.303860 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.303861 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.304057 (+ 196us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.304340 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304343 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.304345 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.304392 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.304797 (+ 405us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.304800 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.304878 (+ 78us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.305614 (+ 736us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.306000 (+ 386us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":203,"spinlock_wait_cycles":32640,"thread_start_us":110,"threads_started":1}
I20260501 02:03:19.306728 26715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.292832 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.18.96.194:46759)
0501 02:03:19.293290 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.293333 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.294505 (+ 1172us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.295002 (+ 497us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.295009 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.295543 (+ 534us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.296021 (+ 478us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.296031 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.297019 (+ 988us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.297022 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.297602 (+ 580us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.297614 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.298096 (+ 482us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.298725 (+ 629us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.298745 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.300829 (+ 2084us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.303621 (+ 2792us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.303645 (+ 24us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.303651 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.303896 (+ 245us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.304169 (+ 273us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304174 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.304179 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.304237 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.304674 (+ 437us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.304680 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.304789 (+ 109us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.305610 (+ 821us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.305855 (+ 245us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":19072,"thread_start_us":110,"threads_started":1}
I20260501 02:03:19.307189 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:38149
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:37935
--webserver_port=41979
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:03:19.306725 26717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.293751 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:55393)
0501 02:03:19.294071 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.294086 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.294516 (+ 430us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.295002 (+ 486us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.295008 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.295538 (+ 530us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.295961 (+ 423us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.295971 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.296948 (+ 977us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.296951 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.297475 (+ 524us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.297482 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.298013 (+ 531us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.298565 (+ 552us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.298600 (+ 35us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.301024 (+ 2424us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.303352 (+ 2328us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.303358 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.303369 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.303703 (+ 334us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.303995 (+ 292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.304000 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.304003 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.304112 (+ 109us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.304535 (+ 423us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.304540 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.304783 (+ 243us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.305785 (+ 1002us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.305921 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"spinlock_wait_cycles":13440,"thread_start_us":83,"threads_started":1}
I20260501 02:03:19.308128 26713 heartbeater.cc:344] Connected to a master server at 127.18.96.253:35673
I20260501 02:03:19.308151 26711 heartbeater.cc:344] Connected to a master server at 127.18.96.252:39139
I20260501 02:03:19.308290 26712 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:19.308347 26711 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.308351 26712 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.308348 26713 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.308910 26712 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
I20260501 02:03:19.308908 26713 heartbeater.cc:507] Master 127.18.96.253:35673 requested a full tablet report, sending...
I20260501 02:03:19.309223 26711 heartbeater.cc:507] Master 127.18.96.252:39139 requested a full tablet report, sending...
I20260501 02:03:19.310070 26196 ts_manager.cc:194] Registered new tserver with Master: b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:19.310071 26379 ts_manager.cc:194] Registered new tserver with Master: b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:19.310132 25586 ts_manager.cc:194] Re-registered known tserver with Master: b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:19.310738 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:55393
W20260501 02:03:19.405743 26718 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:19.405925 26718 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:19.406044 26718 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:19.409121 26718 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:19.409168 26718 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:19.409240 26718 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:19.413196 26718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:38149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=41979
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:37935,127.18.96.253:35673,127.18.96.252:39139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:19.414121 26718 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:19.414885 26718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:19.420667 26723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:19.420715 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:19.420686 26724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:19.420918 26718 server_base.cc:1061] running on GCE node
I20260501 02:03:19.421247 26718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:19.421743 26718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:19.422904 26718 hybrid_clock.cc:648] HybridClock initialized: now 1777600999422869 us; error 56 us; skew 500 ppm
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:19.425498 26718 init.cc:377] Logged in from keytab as oryx/127.18.96.195@KRBTEST.COM (short username oryx)
I20260501 02:03:19.426437 26718 webserver.cc:492] Webserver started at http://127.18.96.195:41979/ using document root <none> and password file <none>
I20260501 02:03:19.426888 26718 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:19.426942 26718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:19.430043 26718 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:03:19.431725 26733 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.432580 26718 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:19.432680 26718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "4962797c6d7f432bac3c64963011b6c1"
format_stamp: "Formatted at 2026-05-01 02:03:17 on dist-test-slave-2j90"
server_key: "4526cd47e05f1122e2e80ef6bc4bd1dc"
server_key_iv: "5779887b5d3858c0b76461efba7ef0e6"
server_key_version: "encryptionkey@0"
I20260501 02:03:19.432998 26718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:19.454941 26718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:19.457180 26718 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:19.457361 26718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:19.457857 26718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:19.458691 26718 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:19.458736 26718 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.458808 26718 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:19.458842 26718 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:19.467860 26718 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:38149
I20260501 02:03:19.467921 26846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:38149 every 8 connection(s)
I20260501 02:03:19.468900 26718 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:19.473426 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26718
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.252@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
May 01 02:03:19 dist-test-slave-2j90 krb5kdc[25538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777600999, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.253@KRBTEST.COM
I20260501 02:03:19.482451 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.470871 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:52909 (local address 127.18.96.252:39139)
0501 02:03:19.470999 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.471003 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.471837 (+ 834us) server_negotiation.cc:408] Connection header received
0501 02:03:19.472214 (+ 377us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.472218 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.472276 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.472349 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.473635 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474400 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.475502 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475689 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.478837 (+ 3148us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.478853 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.478856 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.478882 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.480638 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481109 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.481113 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.481115 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.481162 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481482 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.481487 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.481488 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.481681 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.481764 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.482153 (+ 389us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.482260 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:03:19.483032 26851 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.470683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:35673 (local address 127.18.96.195:59855)
0501 02:03:19.471102 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.471138 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.472062 (+ 924us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.472489 (+ 427us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.472500 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.472949 (+ 449us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.473466 (+ 517us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.473480 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474538 (+ 1058us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.474541 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.475083 (+ 542us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.475090 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475278 (+ 188us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.476313 (+ 1035us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.476329 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.478671 (+ 2342us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.481579 (+ 2908us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481582 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.481584 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.481776 (+ 192us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.482063 (+ 287us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.482067 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.482069 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.482116 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.482517 (+ 401us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.482520 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.482589 (+ 69us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.482694 (+ 105us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.482805 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"spinlock_wait_cycles":19328,"thread_start_us":113,"threads_started":1}
I20260501 02:03:19.483109 26853 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.470683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:39139 (local address 127.18.96.195:52909)
0501 02:03:19.471749 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.471763 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.472047 (+ 284us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.472487 (+ 440us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.472494 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.472943 (+ 449us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.473469 (+ 526us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.473483 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474544 (+ 1061us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.474547 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.475080 (+ 533us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.475088 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475276 (+ 188us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.475835 (+ 559us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.475851 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.478703 (+ 2852us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.480764 (+ 2061us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.480768 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.480770 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.480998 (+ 228us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.481280 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481285 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.481287 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.481368 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.481843 (+ 475us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.481848 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.481938 (+ 90us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.482697 (+ 759us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.482870 (+ 173us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":83,"spinlock_wait_cycles":35328,"thread_start_us":138,"threads_started":1}
I20260501 02:03:19.483489 26441 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.470450 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:59855 (local address 127.18.96.253:35673)
0501 02:03:19.470594 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.470598 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.471229 (+ 631us) server_negotiation.cc:408] Connection header received
0501 02:03:19.472244 (+ 1015us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.472248 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.472301 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.472363 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.473646 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474396 (+ 750us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.475207 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475386 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.478814 (+ 3428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.478836 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.478843 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.478874 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.481470 (+ 2596us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481887 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.481891 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.481892 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.481939 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.482210 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.482216 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.482220 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.482387 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.482475 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.483199 (+ 724us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.483309 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:03:19.483820 26443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.471631 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:34901 (local address 127.18.96.254:37935)
0501 02:03:19.471750 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.471756 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.471881 (+ 125us) server_negotiation.cc:408] Connection header received
0501 02:03:19.472245 (+ 364us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.472249 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.472301 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.472354 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.473817 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474552 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.475312 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475520 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.478320 (+ 2800us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.478338 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.478341 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.478369 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.480232 (+ 1863us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.480870 (+ 638us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.480874 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.480876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.480929 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481270 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.481274 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.481278 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.481449 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.481543 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.483525 (+ 1982us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.483674 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:03:19.483018 26852 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.471495 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:34901)
0501 02:03:19.471814 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.471826 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.472073 (+ 247us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.472489 (+ 416us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.472500 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.473417 (+ 917us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:19.473683 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.473690 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.474677 (+ 987us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.474680 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.475198 (+ 518us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.475204 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.475649 (+ 445us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.476082 (+ 433us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:19.476100 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:19.478159 (+ 2059us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:19.480366 (+ 2207us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.480373 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.480384 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.480717 (+ 333us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.481035 (+ 318us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:19.481039 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:19.481042 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:19.481156 (+ 114us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:19.481601 (+ 445us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:19.481606 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:19.481828 (+ 222us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.482055 (+ 227us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.482263 (+ 208us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":768,"thread_start_us":102,"threads_started":1}
I20260501 02:03:19.484433 26847 heartbeater.cc:344] Connected to a master server at 127.18.96.252:39139
I20260501 02:03:19.484666 26847 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.485219 26847 heartbeater.cc:507] Master 127.18.96.252:39139 requested a full tablet report, sending...
I20260501 02:03:19.485832 26848 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37935
I20260501 02:03:19.485841 26849 heartbeater.cc:344] Connected to a master server at 127.18.96.253:35673
I20260501 02:03:19.485930 26848 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.485944 26849 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:19.486153 26848 heartbeater.cc:507] Master 127.18.96.254:37935 requested a full tablet report, sending...
I20260501 02:03:19.486173 26849 heartbeater.cc:507] Master 127.18.96.253:35673 requested a full tablet report, sending...
I20260501 02:03:19.486317 26379 ts_manager.cc:194] Registered new tserver with Master: 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149)
I20260501 02:03:19.486768 26860 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.477092 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.18.96.254:37935)
0501 02:03:19.477338 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.477342 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.477419 (+ 77us) server_negotiation.cc:408] Connection header received
0501 02:03:19.477538 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.477542 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.477596 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.477790 (+ 194us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.478990 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.479770 (+ 780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.480948 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.481147 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.482259 (+ 1112us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.482275 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.482278 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.482300 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.484813 (+ 2513us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.485252 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.485258 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.485263 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.485318 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.485615 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.485621 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.485623 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.485789 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.485912 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.486458 (+ 546us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.486606 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260501 02:03:19.486856 25586 ts_manager.cc:194] Re-registered known tserver with Master: 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149)
I20260501 02:03:19.486919 26196 ts_manager.cc:194] Registered new tserver with Master: 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149)
I20260501 02:03:19.487501 25586 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:34901
I20260501 02:03:19.492822 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.476498 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.18.96.252:39139)
0501 02:03:19.482630 (+ 6132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.482636 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.482653 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:03:19.484995 (+ 2342us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.484998 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.485045 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.485124 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.486878 (+ 1754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.487439 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.488209 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.488420 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.489359 (+ 939us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.489373 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.489376 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.489399 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.491195 (+ 1796us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.491735 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.491738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.491740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.491782 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.492028 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.492032 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.492035 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.492184 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.492267 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.492601 (+ 334us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.492720 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6058,"thread_start_us":100,"threads_started":1}
I20260501 02:03:19.491144 25586 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:19.494304 25586 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:19.495154 26861 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.477435 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.18.96.253:35673)
0501 02:03:19.477672 (+ 237us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.479202 (+ 1530us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.485351 (+ 6149us) server_negotiation.cc:408] Connection header received
0501 02:03:19.485390 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.485393 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.485437 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.485511 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.488833 (+ 3322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.489473 (+ 640us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.490380 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.490550 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.491739 (+ 1189us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.491753 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.491756 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.491777 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.493575 (+ 1798us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.494079 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.494082 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.494083 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.494155 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.494440 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.494443 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.494445 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.494571 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.494702 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.494922 (+ 220us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.495018 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260501 02:03:19.510597 26873 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.506213 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38149 (local address 127.0.0.1:48802)
0501 02:03:19.506491 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.506506 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.506604 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.507739 (+ 1135us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.507742 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.507756 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:19.508019 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.508028 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.509356 (+ 1328us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.509360 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.510273 (+ 913us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.510280 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.510348 (+ 68us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.510363 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.510399 (+ 36us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.510446 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"spinlock_wait_cycles":384,"thread_start_us":83,"threads_started":1}
I20260501 02:03:19.510807 26874 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.506503 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34991 (local address 127.0.0.1:43508)
0501 02:03:19.506719 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.506734 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.506818 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.507697 (+ 879us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.507703 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.507727 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:19.507993 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.508002 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.508899 (+ 897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.508903 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.509698 (+ 795us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.509707 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.510535 (+ 828us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.510558 (+ 23us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.510638 (+ 80us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.510700 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260501 02:03:19.510900 26875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.506893 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.18.96.194:34991)
0501 02:03:19.507253 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.507262 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.507286 (+ 24us) server_negotiation.cc:408] Connection header received
0501 02:03:19.507385 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.507391 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.507563 (+ 172us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.507711 (+ 148us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:19.508112 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.508791 (+ 679us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.509873 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.510406 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.510484 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.510689 (+ 205us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.510764 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":96,"threads_started":1}
I20260501 02:03:19.511307 26876 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.507189 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45295 (local address 127.0.0.1:49342)
0501 02:03:19.507695 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.507708 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.507777 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.508489 (+ 712us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.508492 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.508510 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:19.508738 (+ 228us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.508744 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.510121 (+ 1377us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.510124 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.510982 (+ 858us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.510992 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.511090 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.511103 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.511154 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.511203 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":91,"threads_started":1}
I20260501 02:03:19.511981 26872 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.506202 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.18.96.195:38149)
0501 02:03:19.507265 (+ 1063us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.507271 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.507289 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:03:19.507369 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.507374 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.507593 (+ 219us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.507902 (+ 309us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:19.508147 (+ 245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.509223 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.511127 (+ 1904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.511629 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.511722 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.511780 (+ 58us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.511849 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":99,"threads_started":1}
I20260501 02:03:19.512349 26877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.507323 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.18.96.193:45295)
0501 02:03:19.507986 (+ 663us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.507991 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.508009 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:03:19.508085 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.508093 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.508233 (+ 140us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.508367 (+ 134us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:19.508856 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.509976 (+ 1120us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.511526 (+ 1550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.511994 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.512090 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.512145 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.512214 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":96,"threads_started":1}
I20260501 02:03:19.513545 26645 tablet_service.cc:1511] Processing CreateTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 (DEFAULT_TABLE table=test-table [id=e3b67936950b45bb8ce4764b0979b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:19.514434 26645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5dfa9b4fe904f4d819a09b34046ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:19.514420 26509 tablet_service.cc:1511] Processing CreateTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 (DEFAULT_TABLE table=test-table [id=e3b67936950b45bb8ce4764b0979b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:19.514645 26781 tablet_service.cc:1511] Processing CreateTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 (DEFAULT_TABLE table=test-table [id=e3b67936950b45bb8ce4764b0979b166]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:19.515314 26781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5dfa9b4fe904f4d819a09b34046ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:19.515309 26509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5dfa9b4fe904f4d819a09b34046ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:19.520305 26878 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Bootstrap starting.
I20260501 02:03:19.521466 26880 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Bootstrap starting.
I20260501 02:03:19.521824 26879 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Bootstrap starting.
I20260501 02:03:19.523274 26878 tablet_bootstrap.cc:654] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:19.523875 26879 tablet_bootstrap.cc:654] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:19.524076 26878 log.cc:826] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:19.524627 26879 log.cc:826] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:19.525889 26878 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: No bootstrap required, opened a new log
I20260501 02:03:19.526093 26878 ts_tablet_manager.cc:1403] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260501 02:03:19.526279 26880 tablet_bootstrap.cc:654] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:19.526315 26879 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: No bootstrap required, opened a new log
I20260501 02:03:19.526482 26879 ts_tablet_manager.cc:1403] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:03:19.527024 26880 log.cc:826] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:19.528793 26880 tablet_bootstrap.cc:492] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: No bootstrap required, opened a new log
I20260501 02:03:19.528985 26880 ts_tablet_manager.cc:1403] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260501 02:03:19.529263 26879 raft_consensus.cc:359] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.529263 26878 raft_consensus.cc:359] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.529446 26878 raft_consensus.cc:385] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:19.529446 26879 raft_consensus.cc:385] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:19.529517 26879 raft_consensus.cc:740] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962797c6d7f432bac3c64963011b6c1, State: Initialized, Role: FOLLOWER
I20260501 02:03:19.529548 26878 raft_consensus.cc:740] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12a7cdcf85747b09cd0afc131f2ece0, State: Initialized, Role: FOLLOWER
I20260501 02:03:19.529991 26879 consensus_queue.cc:260] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.529973 26878 consensus_queue.cc:260] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.530563 26712 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
I20260501 02:03:19.530781 26848 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
I20260501 02:03:19.531544 26879 ts_tablet_manager.cc:1434] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:03:19.532224 26880 raft_consensus.cc:359] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.532452 26880 raft_consensus.cc:385] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:19.532536 26880 raft_consensus.cc:740] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7015c4d1c65476abc167fa1a9733c30, State: Initialized, Role: FOLLOWER
I20260501 02:03:19.532965 26880 consensus_queue.cc:260] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.535146 26576 heartbeater.cc:499] Master 127.18.96.254:37935 was elected leader, sending a full tablet report...
I20260501 02:03:19.535468 26880 ts_tablet_manager.cc:1434] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260501 02:03:19.537861 26878 ts_tablet_manager.cc:1434] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
W20260501 02:03:19.542007 26714 tablet.cc:2404] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:03:19.610769 26578 tablet.cc:2404] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:19.616292 26886 raft_consensus.cc:493] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:19.616429 26886 raft_consensus.cc:515] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.617421 26886 leader_election.cc:290] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149), b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:19.620114 26580 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.617627 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38149 (local address 127.18.96.193:32941)
0501 02:03:19.617753 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.617769 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.617859 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.618153 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.618157 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.618178 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:19.618451 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.618458 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.619251 (+ 793us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.619255 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.619860 (+ 605us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.619866 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.619951 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.619963 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.619996 (+ 33us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.620031 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260501 02:03:19.620500 26579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.617673 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34991 (local address 127.18.96.193:56657)
0501 02:03:19.617805 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:19.617816 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:19.617864 (+ 48us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:19.618388 (+ 524us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:19.618390 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:19.618407 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:19.618614 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.618620 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.619357 (+ 737us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.619360 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:19.620153 (+ 793us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:19.620161 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.620261 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.620273 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:19.620317 (+ 44us) client_negotiation.cc:241] Negotiation successful
0501 02:03:19.620379 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260501 02:03:19.620684 26872 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.617748 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:32941 (local address 127.18.96.195:38149)
0501 02:03:19.617867 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.617872 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.617889 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:03:19.617948 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.617952 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.618033 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.618116 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:19.618570 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.619139 (+ 569us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.619965 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.620416 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.620449 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.620494 (+ 45us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.620551 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260501 02:03:19.621120 26875 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.618004 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:56657 (local address 127.18.96.194:34991)
0501 02:03:19.618160 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.618164 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.618177 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:19.618222 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.618225 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.618274 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.618355 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:19.618730 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.619246 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.620409 (+ 1163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.620902 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.620931 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.620972 (+ 41us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.621025 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:03:19.621258 26801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5dfa9b4fe904f4d819a09b34046ad93" candidate_uuid: "e7015c4d1c65476abc167fa1a9733c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962797c6d7f432bac3c64963011b6c1" is_pre_election: true
I20260501 02:03:19.621542 26801 raft_consensus.cc:2468] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7015c4d1c65476abc167fa1a9733c30 in term 0.
I20260501 02:03:19.621685 26665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5dfa9b4fe904f4d819a09b34046ad93" candidate_uuid: "e7015c4d1c65476abc167fa1a9733c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" is_pre_election: true
I20260501 02:03:19.621917 26665 raft_consensus.cc:2468] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7015c4d1c65476abc167fa1a9733c30 in term 0.
I20260501 02:03:19.622043 26464 leader_election.cc:304] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4962797c6d7f432bac3c64963011b6c1, e7015c4d1c65476abc167fa1a9733c30; no voters:
I20260501 02:03:19.622274 26886 raft_consensus.cc:2804] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:19.622361 26886 raft_consensus.cc:493] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:19.622426 26886 raft_consensus.cc:3060] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:19.623461 26886 raft_consensus.cc:515] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.623862 26886 leader_election.cc:290] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [CANDIDATE]: Term 1 election: Requested vote from peers 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149), b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991)
I20260501 02:03:19.624117 26801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5dfa9b4fe904f4d819a09b34046ad93" candidate_uuid: "e7015c4d1c65476abc167fa1a9733c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962797c6d7f432bac3c64963011b6c1"
I20260501 02:03:19.624189 26665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5dfa9b4fe904f4d819a09b34046ad93" candidate_uuid: "e7015c4d1c65476abc167fa1a9733c30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b12a7cdcf85747b09cd0afc131f2ece0"
I20260501 02:03:19.624236 26801 raft_consensus.cc:3060] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:19.624312 26665 raft_consensus.cc:3060] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:19.625147 26801 raft_consensus.cc:2468] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7015c4d1c65476abc167fa1a9733c30 in term 1.
I20260501 02:03:19.625170 26665 raft_consensus.cc:2468] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7015c4d1c65476abc167fa1a9733c30 in term 1.
I20260501 02:03:19.625458 26462 leader_election.cc:304] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b12a7cdcf85747b09cd0afc131f2ece0, e7015c4d1c65476abc167fa1a9733c30; no voters:
I20260501 02:03:19.625612 26886 raft_consensus.cc:2804] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:19.625821 26886 raft_consensus.cc:697] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 1 LEADER]: Becoming Leader. State: Replica: e7015c4d1c65476abc167fa1a9733c30, State: Running, Role: LEADER
I20260501 02:03:19.626166 26886 consensus_queue.cc:237] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:19.629292 25586 catalog_manager.cc:5671] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "e7015c4d1c65476abc167fa1a9733c30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } health_report { overall_health: HEALTHY } } }
I20260501 02:03:19.660307 26877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.657525 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.18.96.193:45295)
0501 02:03:19.657688 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.657693 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.657706 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:19.657825 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.657828 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.657877 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.657992 (+ 115us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:19.658314 (+ 322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.658765 (+ 451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.659376 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.659552 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.659625 (+ 73us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:19.659949 (+ 324us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:19.660063 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.660133 (+ 70us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.660178 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:03:19.665503 26801 raft_consensus.cc:1275] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Refusing update from remote peer e7015c4d1c65476abc167fa1a9733c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:19.665503 26665 raft_consensus.cc:1275] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Refusing update from remote peer e7015c4d1c65476abc167fa1a9733c30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:19.666229 26886 consensus_queue.cc:1048] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:19.666409 26887 consensus_queue.cc:1048] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:19.673614 26890 mvcc.cc:204] Tried to move back new op lower bound from 7281053694623055872 to 7281053694471266304. Current Snapshot: MvccSnapshot[applied={T|T < 7281053694623055872}]
I20260501 02:03:19.674906 26891 mvcc.cc:204] Tried to move back new op lower bound from 7281053694623055872 to 7281053694471266304. Current Snapshot: MvccSnapshot[applied={T|T < 7281053694623055872}]
I20260501 02:03:19.677340 26892 mvcc.cc:204] Tried to move back new op lower bound from 7281053694623055872 to 7281053694471266304. Current Snapshot: MvccSnapshot[applied={T|T < 7281053694623055872}]
I20260501 02:03:19.682514 25585 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:19.682657 25585 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:19.686573 25585 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 71f3c8b755f34d6a8e7779286445a80f: Sending DeleteTablet for 3 replicas of tablet b5dfa9b4fe904f4d819a09b34046ad93
I20260501 02:03:19.687392 26781 tablet_service.cc:1558] Processing DeleteTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:19 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:48802
I20260501 02:03:19.687855 26900 tablet_replica.cc:333] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: stopping tablet replica
I20260501 02:03:19.688194 26900 raft_consensus.cc:2243] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:19.688532 26900 raft_consensus.cc:2272] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:19.689328 26509 tablet_service.cc:1558] Processing DeleteTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:19 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:49342
I20260501 02:03:19.689800 26645 tablet_service.cc:1558] Processing DeleteTablet for tablet b5dfa9b4fe904f4d819a09b34046ad93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:19 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:43508
I20260501 02:03:19.689899 26900 ts_tablet_manager.cc:1916] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:19.690373 26902 tablet_replica.cc:333] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: stopping tablet replica
I20260501 02:03:19.690632 26902 raft_consensus.cc:2243] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:19.690935 26902 raft_consensus.cc:2272] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:19.691084 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 25554
I20260501 02:03:19.692656 26902 ts_tablet_manager.cc:1916] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:19.693485 26900 ts_tablet_manager.cc:1929] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:19.693652 26900 log.cc:1199] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/b5dfa9b4fe904f4d819a09b34046ad93
I20260501 02:03:19.694072 26900 ts_tablet_manager.cc:1950] T b5dfa9b4fe904f4d819a09b34046ad93 P 4962797c6d7f432bac3c64963011b6c1: Deleting consensus metadata
I20260501 02:03:19.695803 26902 ts_tablet_manager.cc:1929] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:19.695894 26902 log.cc:1199] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/b5dfa9b4fe904f4d819a09b34046ad93
I20260501 02:03:19.696265 26902 ts_tablet_manager.cc:1950] T b5dfa9b4fe904f4d819a09b34046ad93 P b12a7cdcf85747b09cd0afc131f2ece0: Deleting consensus metadata
W20260501 02:03:19.698758 26598 connection.cc:570] server connection from 127.0.0.1:43508 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 02:03:19.699074 26734 connection.cc:570] server connection from 127.0.0.1:48802 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 02:03:19.700028 26901 tablet_replica.cc:333] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: stopping tablet replica
I20260501 02:03:19.700258 26901 raft_consensus.cc:2243] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:19.700623 26901 raft_consensus.cc:2272] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:19.701927 26901 ts_tablet_manager.cc:1916] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:19.704227 26901 ts_tablet_manager.cc:1929] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:19.704320 26901 log.cc:1199] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/b5dfa9b4fe904f4d819a09b34046ad93
I20260501 02:03:19.704619 26901 ts_tablet_manager.cc:1950] T b5dfa9b4fe904f4d819a09b34046ad93 P e7015c4d1c65476abc167fa1a9733c30: Deleting consensus metadata
W20260501 02:03:19.705461 26463 connection.cc:463] server connection from 127.0.0.1:49342 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49342 (request call id 1) could send its response
I20260501 02:03:19.709615 26444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.701702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.18.96.252:39139)
0501 02:03:19.701819 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.701823 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.702076 (+ 253us) server_negotiation.cc:408] Connection header received
0501 02:03:19.702250 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.702254 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.702322 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.702407 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.703453 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.704242 (+ 789us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.705151 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.705340 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.706817 (+ 1477us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.706831 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.706834 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.706854 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.708295 (+ 1441us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.708701 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.708703 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.708705 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.708741 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.708942 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.708944 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.708945 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.709059 (+ 114us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.709150 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.709368 (+ 218us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.709447 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:03:19.716821 26861 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:19.710490 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.18.96.253:35673)
0501 02:03:19.710622 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:19.710626 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:19.710639 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:19.710678 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:19.710681 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:19.710724 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:19.710806 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:19.711426 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.712185 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:19.712760 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:19.712958 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:19.713786 (+ 828us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:19.713803 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:19.713807 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:19.713829 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:19.715446 (+ 1617us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.715831 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.715835 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.715837 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.715895 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:19.716107 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:19.716110 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:19.716112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:19.716236 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:19.716358 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:19.716554 (+ 196us) server_negotiation.cc:300] Negotiation successful
0501 02:03:19.716667 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:03:20.677867 26908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:20.677505 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:36709)
0501 02:03:20.677723 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:20.677782 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
W20260501 02:03:20.678144 26712 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:37935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:20.679368 26909 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:20.678966 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:38407)
0501 02:03:20.679192 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:20.679259 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
W20260501 02:03:20.679621 26576 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:37935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:20.682662 26910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:20.682296 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:34131)
0501 02:03:20.682508 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:20.682571 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":51,"threads_started":1}
W20260501 02:03:20.682894 26848 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:37935 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.187292 26911 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f3c8b755f34d6a8e7779286445a80f)
I20260501 02:03:21.187413 26912 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 71f3c8b755f34d6a8e7779286445a80f)
I20260501 02:03:21.187485 26911 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:21.187628 26912 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:21.188575 26911 leader_election.cc:290] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253:35673)
I20260501 02:03:21.188586 26912 leader_election.cc:290] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139)
I20260501 02:03:21.189136 26913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.188733 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45156)
0501 02:03:21.188945 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.189026 (+ 81us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260501 02:03:21.189229 26389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" is_pre_election: true
I20260501 02:03:21.189244 26206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" is_pre_election: true
I20260501 02:03:21.189392 26914 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.188890 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45166)
0501 02:03:21.189164 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.189223 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":62,"threads_started":1}
I20260501 02:03:21.189424 26389 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2ec88cb58f42ed9c288ef50abc1b42 in term 1.
I20260501 02:03:21.189437 26206 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2ee37aa6a7f422a96e94e8d5abd3bfc in term 1.
W20260501 02:03:21.189522 26181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
W20260501 02:03:21.189744 26364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.190186 26364 leader_election.cc:304] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2ec88cb58f42ed9c288ef50abc1b42, b2ee37aa6a7f422a96e94e8d5abd3bfc; no voters:
I20260501 02:03:21.190184 26180 leader_election.cc:304] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2ec88cb58f42ed9c288ef50abc1b42, b2ee37aa6a7f422a96e94e8d5abd3bfc; no voters:
I20260501 02:03:21.190384 26911 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 02:03:21.190366 26912 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 02:03:21.190436 26911 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71f3c8b755f34d6a8e7779286445a80f)
I20260501 02:03:21.190349 26914 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.190125 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45174)
0501 02:03:21.190244 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.190289 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260501 02:03:21.190469 26912 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 71f3c8b755f34d6a8e7779286445a80f)
I20260501 02:03:21.190483 26911 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:03:21.190519 26912 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:03:21.191421 26911 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:21.191439 26912 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
W20260501 02:03:21.191606 26181 leader_election.cc:336] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.191802 26912 leader_election.cc:290] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 election: Requested vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139)
I20260501 02:03:21.191825 26911 leader_election.cc:290] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 election: Requested vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), 6b2ec88cb58f42ed9c288ef50abc1b42 (127.18.96.253:35673)
I20260501 02:03:21.192222 26913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.192025 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45184)
0501 02:03:21.192117 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.192164 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260501 02:03:21.192222 26914 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.192000 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45178)
0501 02:03:21.192133 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.192169 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260501 02:03:21.192315 26364 leader_election.cc:336] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.192363 26206 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42"
I20260501 02:03:21.192510 26206 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b2ee37aa6a7f422a96e94e8d5abd3bfc in current term 2: Already voted for candidate 6b2ec88cb58f42ed9c288ef50abc1b42 in this term.
I20260501 02:03:21.192550 26389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
I20260501 02:03:21.192690 26389 raft_consensus.cc:2393] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b2ec88cb58f42ed9c288ef50abc1b42 in current term 2: Already voted for candidate b2ee37aa6a7f422a96e94e8d5abd3bfc in this term.
I20260501 02:03:21.192981 26914 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.192679 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45200)
0501 02:03:21.192889 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.192929 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260501 02:03:21.193094 26181 leader_election.cc:336] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.193171 26913 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.192973 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:45206)
0501 02:03:21.193062 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.193104 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260501 02:03:21.193181 26181 leader_election.cc:304] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b2ec88cb58f42ed9c288ef50abc1b42; no voters: 71f3c8b755f34d6a8e7779286445a80f, b2ee37aa6a7f422a96e94e8d5abd3bfc
I20260501 02:03:21.193316 26912 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260501 02:03:21.193456 26364 leader_election.cc:336] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:21.193542 26364 leader_election.cc:304] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2ee37aa6a7f422a96e94e8d5abd3bfc; no voters: 6b2ec88cb58f42ed9c288ef50abc1b42, 71f3c8b755f34d6a8e7779286445a80f
I20260501 02:03:21.193665 26911 raft_consensus.cc:2749] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260501 02:03:21.679114 26917 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.678715 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:46883)
0501 02:03:21.678947 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.678999 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260501 02:03:21.680406 26918 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.680033 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:51041)
0501 02:03:21.680264 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.680314 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260501 02:03:21.683724 26919 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:21.683364 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:34413)
0501 02:03:21.683579 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:21.683633 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260501 02:03:22.680079 26920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:22.679734 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:48163)
0501 02:03:22.679946 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:22.679992 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260501 02:03:22.681458 26921 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:22.681110 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:45545)
0501 02:03:22.681307 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:22.681360 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260501 02:03:22.684929 26922 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:22.684575 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:51933)
0501 02:03:22.684783 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:22.684826 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260501 02:03:23.341049 26923 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:23.341166 26923 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:23.341517 26923 leader_election.cc:290] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139)
I20260501 02:03:23.341894 26389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" is_pre_election: true
I20260501 02:03:23.342093 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.341700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:40772)
0501 02:03:23.341907 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.341990 (+ 83us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260501 02:03:23.342087 26389 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2ec88cb58f42ed9c288ef50abc1b42 in term 2.
I20260501 02:03:23.342419 26180 leader_election.cc:304] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b2ec88cb58f42ed9c288ef50abc1b42, b2ee37aa6a7f422a96e94e8d5abd3bfc; no voters:
I20260501 02:03:23.342587 26923 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260501 02:03:23.342655 26923 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:23.342684 26923 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 2 FOLLOWER]: Advancing to term 3
I20260501 02:03:23.343570 26923 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:23.343858 26923 leader_election.cc:290] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 election: Requested vote from peers 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935), b2ee37aa6a7f422a96e94e8d5abd3bfc (127.18.96.252:39139)
I20260501 02:03:23.344081 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.343855 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:40776)
0501 02:03:23.343976 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.344023 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260501 02:03:23.344187 26389 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc"
W20260501 02:03:23.344264 26181 leader_election.cc:336] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:23.344313 26389 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 2 FOLLOWER]: Advancing to term 3
I20260501 02:03:23.344758 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.344528 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:40780)
0501 02:03:23.344644 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.344686 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260501 02:03:23.344866 26181 leader_election.cc:336] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
I20260501 02:03:23.345355 26389 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2ec88cb58f42ed9c288ef50abc1b42 in term 3.
I20260501 02:03:23.345623 26180 leader_election.cc:304] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b2ec88cb58f42ed9c288ef50abc1b42, b2ee37aa6a7f422a96e94e8d5abd3bfc; no voters: 71f3c8b755f34d6a8e7779286445a80f
I20260501 02:03:23.345775 26923 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 3 FOLLOWER]: Leader election won for term 3
I20260501 02:03:23.345983 26923 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [term 3 LEADER]: Becoming Leader. State: Replica: 6b2ec88cb58f42ed9c288ef50abc1b42, State: Running, Role: LEADER
I20260501 02:03:23.346218 26923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } }
I20260501 02:03:23.347553 26926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2ec88cb58f42ed9c288ef50abc1b42. Latest consensus state: current_term: 3 leader_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:23.347676 26926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:23.347952 26928 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:23.350083 26928 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:23.350450 26928 catalog_manager.cc:1269] Loaded cluster ID: e45cd13ad7af4b85b473412450a56b5b
I20260501 02:03:23.350494 26928 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:23.351013 26928 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:23.351387 26928 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Loaded TSK: 0
I20260501 02:03:23.351919 26928 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:03:23.407492 26195 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:23.409797 26195 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:23.412631 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.412375 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:40804)
0501 02:03:23.412510 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.412559 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260501 02:03:23.412631 26389 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [term 3 FOLLOWER]: Refusing update from remote peer 6b2ec88cb58f42ed9c288ef50abc1b42: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260501 02:03:23.413072 26926 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260501 02:03:23.413303 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.413101 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.0.0.1:40808)
0501 02:03:23.413205 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.413246 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260501 02:03:23.413539 26181 consensus_peers.cc:597] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 -> Peer 71f3c8b755f34d6a8e7779286445a80f (127.18.96.254:37935): Couldn't send request to peer 71f3c8b755f34d6a8e7779286445a80f. Status: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260501 02:03:23.414464 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2ec88cb58f42ed9c288ef50abc1b42. Latest consensus state: current_term: 3 leader_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:23.414587 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:23.415423 26923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:23.415524 26923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:23.415856 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "71f3c8b755f34d6a8e7779286445a80f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37935 } } peers { permanent_uuid: "6b2ec88cb58f42ed9c288ef50abc1b42" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 35673 } attrs { promote: false } } peers { permanent_uuid: "b2ee37aa6a7f422a96e94e8d5abd3bfc" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 39139 } attrs { promote: false } } }
I20260501 02:03:23.415920 26931 mvcc.cc:204] Tried to move back new op lower bound from 7281053709974237184 to 7281053709708505088. Current Snapshot: MvccSnapshot[applied={T|T < 7281053709974237184}]
I20260501 02:03:23.416059 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ee37aa6a7f422a96e94e8d5abd3bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:03:23.423779 26924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.420403 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34991 (local address 127.0.0.1:36954)
0501 02:03:23.420752 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.420767 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:23.420879 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:23.421445 (+ 566us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:23.421447 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:23.421473 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:23.421703 (+ 230us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.421711 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.422568 (+ 857us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.422573 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:23.423421 (+ 848us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.423427 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.423537 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.423553 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:03:23.423604 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:03:23.423663 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260501 02:03:23.423928 26937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.420786 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38149 (local address 127.0.0.1:46410)
0501 02:03:23.420990 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.421002 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:23.421079 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:23.421799 (+ 720us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:23.421805 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:23.421827 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:23.422079 (+ 252us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.422087 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.422822 (+ 735us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.422825 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:23.423596 (+ 771us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.423605 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.423703 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.423718 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:03:23.423765 (+ 47us) client_negotiation.cc:241] Negotiation successful
0501 02:03:23.423813 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260501 02:03:23.424356 26936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.420541 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.18.96.194:34991)
0501 02:03:23.420743 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.420748 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.420836 (+ 88us) server_negotiation.cc:408] Connection header received
0501 02:03:23.421273 (+ 437us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.421282 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.421336 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.421431 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:23.421855 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.422447 (+ 592us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.423539 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.424078 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.424114 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.424164 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.424232 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260501 02:03:23.424929 26645 tablet_service.cc:1511] Processing CreateTablet for tablet 22c41d20f88c4464bdf049145f82c39a (DEFAULT_TABLE table=test-table [id=211a6bab0ac74a2eb6e2de2b58070faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:23.425226 26645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c41d20f88c4464bdf049145f82c39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:23.425336 26939 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.421298 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45295 (local address 127.0.0.1:57070)
0501 02:03:23.422661 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.422672 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:23.422751 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:23.422994 (+ 243us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:23.422997 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:23.423015 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:23.423227 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.423232 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.424194 (+ 962us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.424197 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:23.424987 (+ 790us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.424996 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.425099 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.425111 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:03:23.425161 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:03:23.425210 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":91,"threads_started":1}
I20260501 02:03:23.426222 26938 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.421128 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.18.96.195:38149)
0501 02:03:23.421448 (+ 320us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.421452 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.421467 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:23.421520 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.421523 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.421576 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.421666 (+ 90us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:23.422188 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.422675 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.424159 (+ 1484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.424669 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.426039 (+ 1370us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.426083 (+ 44us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.426133 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":82,"threads_started":1}
I20260501 02:03:23.426218 26940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.421669 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.18.96.193:45295)
0501 02:03:23.421905 (+ 236us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.421908 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.422749 (+ 841us) server_negotiation.cc:408] Connection header received
0501 02:03:23.422843 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.422846 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.422895 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.422973 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:23.423366 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.424073 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.425487 (+ 1414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.425931 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.425998 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.426046 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.426096 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":63,"threads_started":1}
I20260501 02:03:23.426789 26781 tablet_service.cc:1511] Processing CreateTablet for tablet 22c41d20f88c4464bdf049145f82c39a (DEFAULT_TABLE table=test-table [id=211a6bab0ac74a2eb6e2de2b58070faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:23.427075 26781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c41d20f88c4464bdf049145f82c39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:23.427533 26941 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Bootstrap starting.
I20260501 02:03:23.427968 26509 tablet_service.cc:1511] Processing CreateTablet for tablet 22c41d20f88c4464bdf049145f82c39a (DEFAULT_TABLE table=test-table [id=211a6bab0ac74a2eb6e2de2b58070faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:23.428226 26509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22c41d20f88c4464bdf049145f82c39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:23.428632 26941 tablet_bootstrap.cc:654] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:23.429654 26941 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: No bootstrap required, opened a new log
I20260501 02:03:23.429718 26941 ts_tablet_manager.cc:1403] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:23.430130 26941 raft_consensus.cc:359] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.430168 26943 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Bootstrap starting.
I20260501 02:03:23.430217 26941 raft_consensus.cc:385] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:23.430238 26941 raft_consensus.cc:740] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12a7cdcf85747b09cd0afc131f2ece0, State: Initialized, Role: FOLLOWER
I20260501 02:03:23.430348 26941 consensus_queue.cc:260] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.430634 26941 ts_tablet_manager.cc:1434] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:23.431047 26944 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30: Bootstrap starting.
I20260501 02:03:23.431125 26943 tablet_bootstrap.cc:654] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:23.432045 26944 tablet_bootstrap.cc:654] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:23.432394 26947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.431961 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:40555)
0501 02:03:23.432255 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.432304 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260501 02:03:23.433177 26713 heartbeater.cc:499] Master 127.18.96.253:35673 was elected leader, sending a full tablet report...
I20260501 02:03:23.433279 26944 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30: No bootstrap required, opened a new log
I20260501 02:03:23.433346 26944 ts_tablet_manager.cc:1403] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:23.433799 26944 raft_consensus.cc:359] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.433950 26944 raft_consensus.cc:385] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:23.434010 26943 tablet_bootstrap.cc:492] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: No bootstrap required, opened a new log
I20260501 02:03:23.434053 26944 raft_consensus.cc:740] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7015c4d1c65476abc167fa1a9733c30, State: Initialized, Role: FOLLOWER
I20260501 02:03:23.434096 26943 ts_tablet_manager.cc:1403] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260501 02:03:23.434252 26944 consensus_queue.cc:260] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.434556 26944 ts_tablet_manager.cc:1434] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:23.434546 26943 raft_consensus.cc:359] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.434660 26943 raft_consensus.cc:385] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:23.434697 26943 raft_consensus.cc:740] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4962797c6d7f432bac3c64963011b6c1, State: Initialized, Role: FOLLOWER
I20260501 02:03:23.434823 26943 consensus_queue.cc:260] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.435134 26943 ts_tablet_manager.cc:1434] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:23.435454 26951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.434966 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:35891)
0501 02:03:23.435337 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.435380 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":87,"threads_started":1}
I20260501 02:03:23.436033 26952 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.435661 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:58889)
0501 02:03:23.435908 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.435957 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260501 02:03:23.436456 26951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.436275 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:50521)
0501 02:03:23.436362 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.436400 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260501 02:03:23.437004 26952 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.436663 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:45489)
0501 02:03:23.436856 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.436929 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260501 02:03:23.437189 26849 heartbeater.cc:499] Master 127.18.96.253:35673 was elected leader, sending a full tablet report...
W20260501 02:03:23.473682 26850 tablet.cc:2404] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:23.626035 26945 raft_consensus.cc:493] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:23.626168 26945 raft_consensus.cc:515] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.627242 26945 leader_election.cc:290] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149), e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
I20260501 02:03:23.629946 26947 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.627494 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38149 (local address 127.18.96.194:36505)
0501 02:03:23.627657 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.627673 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:23.627783 (+ 110us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:23.628067 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:23.628071 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:23.628091 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:23.628307 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.628312 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.628999 (+ 687us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.629003 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:23.629562 (+ 559us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.629568 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.629706 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.629721 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:03:23.629769 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:03:23.629825 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260501 02:03:23.630237 26953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.627494 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45295 (local address 127.18.96.194:41131)
0501 02:03:23.627932 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.627942 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:23.628038 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:23.628276 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:23.628279 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:23.628296 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:23.628535 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.628540 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.629219 (+ 679us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.629222 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:23.629770 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:23.629813 (+ 43us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.629915 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.629997 (+ 82us) client_negotiation.cc:770] Sending connection context
0501 02:03:23.630049 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:03:23.630093 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":150,"thread_start_us":126,"threads_started":1}
I20260501 02:03:23.630450 26938 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.627599 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:36505 (local address 127.18.96.195:38149)
0501 02:03:23.627715 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.627721 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.627740 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:03:23.627906 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.627913 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.627957 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.628041 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:23.628417 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.628894 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.629687 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.630172 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.630220 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.630266 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.630317 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260501 02:03:23.630686 26940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.627635 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:41131 (local address 127.18.96.193:45295)
0501 02:03:23.627781 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.627785 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.628010 (+ 225us) server_negotiation.cc:408] Connection header received
0501 02:03:23.628140 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.628143 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.628184 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.628266 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:23.628641 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.629089 (+ 448us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.629901 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.630420 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.630459 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.630509 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.630580 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:03:23.630906 26801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c41d20f88c4464bdf049145f82c39a" candidate_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962797c6d7f432bac3c64963011b6c1" is_pre_election: true
I20260501 02:03:23.631058 26801 raft_consensus.cc:2468] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b12a7cdcf85747b09cd0afc131f2ece0 in term 0.
I20260501 02:03:23.631160 26529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c41d20f88c4464bdf049145f82c39a" candidate_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7015c4d1c65476abc167fa1a9733c30" is_pre_election: true
I20260501 02:03:23.631346 26529 raft_consensus.cc:2468] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b12a7cdcf85747b09cd0afc131f2ece0 in term 0.
I20260501 02:03:23.631430 26600 leader_election.cc:304] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4962797c6d7f432bac3c64963011b6c1, b12a7cdcf85747b09cd0afc131f2ece0; no voters:
I20260501 02:03:23.631695 26945 raft_consensus.cc:2804] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:23.631768 26945 raft_consensus.cc:493] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:23.631815 26945 raft_consensus.cc:3060] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:23.632637 26945 raft_consensus.cc:515] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.632915 26945 leader_election.cc:290] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [CANDIDATE]: Term 1 election: Requested vote from peers 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149), e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295)
I20260501 02:03:23.633231 26529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c41d20f88c4464bdf049145f82c39a" candidate_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7015c4d1c65476abc167fa1a9733c30"
I20260501 02:03:23.633276 26801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22c41d20f88c4464bdf049145f82c39a" candidate_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4962797c6d7f432bac3c64963011b6c1"
I20260501 02:03:23.633340 26529 raft_consensus.cc:3060] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:23.633374 26801 raft_consensus.cc:3060] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:23.634261 26529 raft_consensus.cc:2468] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b12a7cdcf85747b09cd0afc131f2ece0 in term 1.
I20260501 02:03:23.634280 26801 raft_consensus.cc:2468] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b12a7cdcf85747b09cd0afc131f2ece0 in term 1.
I20260501 02:03:23.634588 26599 leader_election.cc:304] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b12a7cdcf85747b09cd0afc131f2ece0, e7015c4d1c65476abc167fa1a9733c30; no voters:
I20260501 02:03:23.634786 26945 raft_consensus.cc:2804] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:23.634976 26945 raft_consensus.cc:697] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 LEADER]: Becoming Leader. State: Replica: b12a7cdcf85747b09cd0afc131f2ece0, State: Running, Role: LEADER
I20260501 02:03:23.635373 26945 consensus_queue.cc:237] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } }
I20260501 02:03:23.635587 26953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.635383 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:44875)
0501 02:03:23.635497 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.635543 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260501 02:03:23.638451 26195 catalog_manager.cc:5671] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b12a7cdcf85747b09cd0afc131f2ece0" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 } health_report { overall_health: UNKNOWN } } }
I20260501 02:03:23.658506 26936 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.655439 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.18.96.194:34991)
0501 02:03:23.655565 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:23.655569 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:23.655588 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:03:23.655790 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:23.655793 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:23.655843 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:23.655923 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:23.656270 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.656728 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:23.657547 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:23.657683 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:23.657724 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:23.658145 (+ 421us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:23.658227 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:23.658321 (+ 94us) server_negotiation.cc:300] Negotiation successful
0501 02:03:23.658358 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260501 02:03:23.660930 26801 raft_consensus.cc:1275] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Refusing update from remote peer b12a7cdcf85747b09cd0afc131f2ece0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:23.661146 26529 raft_consensus.cc:1275] T 22c41d20f88c4464bdf049145f82c39a P e7015c4d1c65476abc167fa1a9733c30 [term 1 FOLLOWER]: Refusing update from remote peer b12a7cdcf85747b09cd0afc131f2ece0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:23.661363 26945 consensus_queue.cc:1048] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4962797c6d7f432bac3c64963011b6c1" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:23.661648 26954 consensus_queue.cc:1048] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7015c4d1c65476abc167fa1a9733c30" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:23.662951 26952 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.662663 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:39457)
0501 02:03:23.662822 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.662878 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260501 02:03:23.664196 26577 heartbeater.cc:499] Master 127.18.96.253:35673 was elected leader, sending a full tablet report...
I20260501 02:03:23.664938 26951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.664689 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:45841)
0501 02:03:23.664807 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.664870 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260501 02:03:23.667905 26952 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.667569 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.195:58685)
0501 02:03:23.667803 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.667846 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260501 02:03:23.670081 26953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.666796 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:56031)
0501 02:03:23.669947 (+ 3151us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.670014 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3089}
I20260501 02:03:23.671262 26951 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.670789 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.193:55379)
0501 02:03:23.671135 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.671193 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260501 02:03:23.672967 26953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.670539 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37935 (local address 127.18.96.194:40481)
0501 02:03:23.672865 (+ 2326us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.672908 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:37935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2259}
I20260501 02:03:23.673717 26195 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:23.673843 26195 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:23.677277 26195 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 6b2ec88cb58f42ed9c288ef50abc1b42: Sending DeleteTablet for 3 replicas of tablet 22c41d20f88c4464bdf049145f82c39a
I20260501 02:03:23.677783 26645 tablet_service.cc:1558] Processing DeleteTablet for tablet 22c41d20f88c4464bdf049145f82c39a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:23 UTC) from {username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:36954
I20260501 02:03:23.678004 26781 tablet_service.cc:1558] Processing DeleteTablet for tablet 22c41d20f88c4464bdf049145f82c39a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:23 UTC) from {username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:46410
I20260501 02:03:23.678098 26967 tablet_replica.cc:333] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: stopping tablet replica
I20260501 02:03:23.678225 26967 raft_consensus.cc:2243] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:23.678323 26509 tablet_service.cc:1558] Processing DeleteTablet for tablet 22c41d20f88c4464bdf049145f82c39a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:23 UTC) from {username='oryx', principal='oryx/127.18.96.253@KRBTEST.COM'} at 127.0.0.1:57070
I20260501 02:03:23.678472 26967 raft_consensus.cc:2272] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:23.679015 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26446
I20260501 02:03:23.679080 26967 ts_tablet_manager.cc:1916] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:23.680928 26967 ts_tablet_manager.cc:1929] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:23.680943 26968 tablet_replica.cc:333] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: stopping tablet replica
I20260501 02:03:23.681011 26967 log.cc:1199] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/22c41d20f88c4464bdf049145f82c39a
I20260501 02:03:23.681066 26968 raft_consensus.cc:2243] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:23.681188 26968 raft_consensus.cc:2272] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:23.681320 26967 ts_tablet_manager.cc:1950] T 22c41d20f88c4464bdf049145f82c39a P b12a7cdcf85747b09cd0afc131f2ece0: Deleting consensus metadata
I20260501 02:03:23.682247 26968 ts_tablet_manager.cc:1916] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:23.683049 26180 catalog_manager.cc:5002] TS b12a7cdcf85747b09cd0afc131f2ece0 (127.18.96.194:34991): tablet 22c41d20f88c4464bdf049145f82c39a (table test-table [id=211a6bab0ac74a2eb6e2de2b58070faa]) successfully deleted
I20260501 02:03:23.684098 26968 ts_tablet_manager.cc:1929] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:23.684190 26968 log.cc:1199] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/22c41d20f88c4464bdf049145f82c39a
I20260501 02:03:23.684484 26968 ts_tablet_manager.cc:1950] T 22c41d20f88c4464bdf049145f82c39a P 4962797c6d7f432bac3c64963011b6c1: Deleting consensus metadata
I20260501 02:03:23.685050 26182 catalog_manager.cc:5002] TS 4962797c6d7f432bac3c64963011b6c1 (127.18.96.195:38149): tablet 22c41d20f88c4464bdf049145f82c39a (table test-table [id=211a6bab0ac74a2eb6e2de2b58070faa]) successfully deleted
I20260501 02:03:23.690727 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26582
I20260501 02:03:23.690768 26937 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:23.690490 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45295 (local address 127.0.0.1:57078)
0501 02:03:23.690618 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:23.690675 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260501 02:03:23.691428 26181 catalog_manager.cc:4729] TS e7015c4d1c65476abc167fa1a9733c30 (127.18.96.193:45295): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22c41d20f88c4464bdf049145f82c39a: Network error: Client connection negotiation failed: client connection to 127.18.96.193:45295: connect: Connection refused (error 111)
I20260501 02:03:23.696834 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26718
I20260501 02:03:23.702253 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26058
I20260501 02:03:23.708165 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26162
2026-05-01T02:03:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9399 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:23 dist-test-slave-2j90 krb5kdc[26972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:23 dist-test-slave-2j90 krb5kdc[26972](info): set up 2 sockets
May 01 02:03:23 dist-test-slave-2j90 krb5kdc[26972](info): commencing operation
krb5kdc: starting...
W20260501 02:03:25.659040 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.928s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:25 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:25.789057 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40755
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:25.885854 26988 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:25.886101 26988 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:25.886181 26988 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:25.889336 26988 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:25.889415 26988 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260501 02:03:25.889447 26988 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:25.889510 26988 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:25.889559 26988 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:25.889596 26988 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:25.893707 26988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40755
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:25.894716 26988 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:25.895512 26988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:25.900791 26993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:25.900803 26996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:25.900866 26988 server_base.cc:1061] running on GCE node
W20260501 02:03:25.900803 26994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:25.901556 26988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:25.902385 26988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:25.903559 26988 hybrid_clock.cc:648] HybridClock initialized: now 1777601005903526 us; error 40 us; skew 500 ppm
May 01 02:03:25 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601005, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:25.906170 26988 init.cc:377] Logged in from keytab as oryx/127.18.96.254@KRBTEST.COM (short username oryx)
I20260501 02:03:25.907173 26988 webserver.cc:492] Webserver started at http://127.18.96.254:46545/ using document root <none> and password file <none>
I20260501 02:03:25.907764 26988 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:25.907862 26988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:25.908165 26988 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:25.909883 26988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "87c24bc67e174de4851812542a372920"
format_stamp: "Formatted at 2026-05-01 02:03:25 on dist-test-slave-2j90"
server_key: "b89c31be2845d91e636ee7608c1471d7"
server_key_iv: "79affb5ca3265a77312b7b9b9a8db629"
server_key_version: "encryptionkey@0"
I20260501 02:03:25.910383 26988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "87c24bc67e174de4851812542a372920"
format_stamp: "Formatted at 2026-05-01 02:03:25 on dist-test-slave-2j90"
server_key: "b89c31be2845d91e636ee7608c1471d7"
server_key_iv: "79affb5ca3265a77312b7b9b9a8db629"
server_key_version: "encryptionkey@0"
I20260501 02:03:25.913406 26988 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:25.915382 27003 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:25.916290 26988 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:03:25.916436 26988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "87c24bc67e174de4851812542a372920"
format_stamp: "Formatted at 2026-05-01 02:03:25 on dist-test-slave-2j90"
server_key: "b89c31be2845d91e636ee7608c1471d7"
server_key_iv: "79affb5ca3265a77312b7b9b9a8db629"
server_key_version: "encryptionkey@0"
I20260501 02:03:25.916532 26988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:25.929947 26988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:25.932241 26988 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:25.932449 26988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:25.940672 26988 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40755
I20260501 02:03:25.940719 27065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40755 every 8 connection(s)
I20260501 02:03:25.941692 26988 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:25.944150 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 26988
I20260501 02:03:25.944254 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:25.944512 18819 external_mini_cluster.cc:1468] Setting key 92b61b94026ff3344944cd4aa63e5bfd
I20260501 02:03:25.944558 27066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:25.950314 27066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Bootstrap starting.
May 01 02:03:25 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:25.953177 27066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:25.953948 27066 log.cc:826] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:25.955710 27066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: No bootstrap required, opened a new log
I20260501 02:03:25.957523 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:25.945689 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.18.96.254:40755)
0501 02:03:25.946126 (+ 437us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:25.946134 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:25.946166 (+ 32us) server_negotiation.cc:408] Connection header received
0501 02:03:25.946968 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:25.946987 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:25.947235 (+ 248us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:25.947475 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:25.948553 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:25.949354 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:25.950262 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:25.950547 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:25.953245 (+ 2698us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:25.953262 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:25.953292 (+ 30us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:25.953330 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:25.955270 (+ 1940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:25.955732 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:25.955736 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:25.955739 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:25.955822 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:25.956050 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:25.956054 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:25.956055 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:25.956320 (+ 265us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:25.956505 (+ 185us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:25.956788 (+ 283us) server_negotiation.cc:300] Negotiation successful
0501 02:03:25.957025 (+ 237us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":165,"threads_started":1}
I20260501 02:03:25.958621 27066 raft_consensus.cc:359] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } }
I20260501 02:03:25.958806 27066 raft_consensus.cc:385] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:25.958889 27066 raft_consensus.cc:740] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c24bc67e174de4851812542a372920, State: Initialized, Role: FOLLOWER
I20260501 02:03:25.959316 27066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } }
I20260501 02:03:25.959460 27066 raft_consensus.cc:399] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:25.959520 27066 raft_consensus.cc:493] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:25.959614 27066 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:25.960372 27066 raft_consensus.cc:515] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } }
I20260501 02:03:25.960686 27066 leader_election.cc:304] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87c24bc67e174de4851812542a372920; no voters:
I20260501 02:03:25.960986 27066 leader_election.cc:290] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:25.961081 27071 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:25.961318 27071 raft_consensus.cc:697] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [term 1 LEADER]: Becoming Leader. State: Replica: 87c24bc67e174de4851812542a372920, State: Running, Role: LEADER
I20260501 02:03:25.961663 27071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } }
I20260501 02:03:25.962110 27066 sys_catalog.cc:565] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:25.963132 27072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87c24bc67e174de4851812542a372920. Latest consensus state: current_term: 1 leader_uuid: "87c24bc67e174de4851812542a372920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } } }
I20260501 02:03:25.963238 27072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:25.963609 27080 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:25.963572 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87c24bc67e174de4851812542a372920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87c24bc67e174de4851812542a372920" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40755 } } }
I20260501 02:03:25.964169 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:25.965926 27080 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:25.971499 27080 catalog_manager.cc:1357] Generated new cluster ID: da4b446a346d4ab08cb6557230f27ef1
I20260501 02:03:25.971560 27080 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:26.022922 27080 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:26.024758 27080 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:26.035378 27080 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Generated new TSK 0
I20260501 02:03:26.035938 27080 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:26.093400 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40755
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:26.190651 27094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:26.190838 27094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:26.190894 27094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:26.194214 27094 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:26.194267 27094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:26.194365 27094 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:03:26.194471 27094 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:26.198383 27094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:26.199236 27094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:26.200009 27094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:26.206269 27102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:26.206380 27094 server_base.cc:1061] running on GCE node
W20260501 02:03:26.206286 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:26.206290 27100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:26.206852 27094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:26.207340 27094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:26.208503 27094 hybrid_clock.cc:648] HybridClock initialized: now 1777601006208480 us; error 38 us; skew 500 ppm
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:26.210877 27094 init.cc:377] Logged in from keytab as oryx/127.18.96.193@KRBTEST.COM (short username oryx)
I20260501 02:03:26.211788 27094 webserver.cc:492] Webserver started at http://127.18.96.193:36759/ using document root <none> and password file <none>
I20260501 02:03:26.212281 27094 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:26.212327 27094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:26.212530 27094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:26.214099 27094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "c578cc17dba44a8eb8115c3edd2878f9"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "6720d56df49bdece7148fd5736edd3a2"
server_key_iv: "78cf62939d4ee7ea2d1390d93b232275"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.214561 27094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "c578cc17dba44a8eb8115c3edd2878f9"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "6720d56df49bdece7148fd5736edd3a2"
server_key_iv: "78cf62939d4ee7ea2d1390d93b232275"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.217459 27094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:26.219403 27109 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.220346 27094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:26.220453 27094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "c578cc17dba44a8eb8115c3edd2878f9"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "6720d56df49bdece7148fd5736edd3a2"
server_key_iv: "78cf62939d4ee7ea2d1390d93b232275"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.220549 27094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:26.250790 27094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:26.253237 27094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:26.253441 27094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:26.254992 27094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:26.255060 27094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.255127 27094 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:26.255172 27094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:26.267069 27094 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:45283
I20260501 02:03:26.267097 27224 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:45283 every 8 connection(s)
I20260501 02:03:26.267577 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.255535 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:34665 (local address 127.18.96.254:40755)
0501 02:03:26.255738 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.255742 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.256414 (+ 672us) server_negotiation.cc:408] Connection header received
0501 02:03:26.257646 (+ 1232us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.257650 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.257698 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.257811 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.259414 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.259961 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.260972 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.261207 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.263576 (+ 2369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.263612 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.263615 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.263645 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.265535 (+ 1890us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.266117 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.266121 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.266123 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.266177 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.266454 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.266460 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.266461 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.266662 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.266773 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.267299 (+ 526us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.267427 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260501 02:03:26.268091 27094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:26.268095 27118 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.255828 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.193:34665)
0501 02:03:26.256249 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.256290 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.257293 (+ 1003us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.258007 (+ 714us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.258020 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.258562 (+ 542us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.259246 (+ 684us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.259264 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.260117 (+ 853us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.260124 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.260842 (+ 718us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.260853 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.261088 (+ 235us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.261716 (+ 628us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.261741 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.263413 (+ 1672us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.265670 (+ 2257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.265676 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.265686 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.266032 (+ 346us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.266275 (+ 243us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.266277 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.266279 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.266356 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.266768 (+ 412us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.266773 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.267039 (+ 266us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.267243 (+ 204us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.267443 (+ 200us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260501 02:03:26.268394 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27094
I20260501 02:03:26.268515 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:26.268815 18819 external_mini_cluster.cc:1468] Setting key 4d0aff47deb1f4e45b62d77d1cc7f988
I20260501 02:03:26.270169 27225 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40755
I20260501 02:03:26.270373 27225 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:26.270831 27225 heartbeater.cc:507] Master 127.18.96.254:40755 requested a full tablet report, sending...
I20260501 02:03:26.272212 27019 ts_manager.cc:194] Registered new tserver with Master: c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283)
I20260501 02:03:26.273241 27019 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:34665
WARNING: no policy specified for oryx/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:03:26.279692 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.271841 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:60859 (local address 127.18.96.254:40755)
0501 02:03:26.271972 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.271979 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.273313 (+ 1334us) server_negotiation.cc:408] Connection header received
0501 02:03:26.273420 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.273423 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.273462 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.273544 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.274263 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.274990 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.275869 (+ 879us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.275999 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.276911 (+ 912us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.276924 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.276926 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.276946 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.278301 (+ 1355us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.278731 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.278733 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.278734 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.278768 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.279015 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.279017 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.279018 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.279123 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.279191 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.279443 (+ 252us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.279526 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:03:26.279731 27231 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.271768 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.193:60859)
0501 02:03:26.273210 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.273226 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.273332 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.273576 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.273580 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.273819 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.274118 (+ 299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.274125 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.275117 (+ 992us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.275124 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.275739 (+ 615us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.275750 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.275883 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.276344 (+ 461us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.276364 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.276789 (+ 425us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.278421 (+ 1632us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.278426 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.278428 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.278637 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.278849 (+ 212us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.278862 (+ 13us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.278866 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.278927 (+ 61us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.279209 (+ 282us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.279214 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.279300 (+ 86us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.279405 (+ 105us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.279522 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:26.315172 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40755
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:26.409356 27236 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:26.409552 27236 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:26.409634 27236 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:26.412716 27236 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:26.412797 27236 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:26.412875 27236 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:03:26.412961 27236 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:26.417721 27236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:26.418923 27236 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:26.419601 27236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:26.425374 27242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:26.425380 27241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:26.425380 27244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:26.425516 27236 server_base.cc:1061] running on GCE node
I20260501 02:03:26.425896 27236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:26.426427 27236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:26.427601 27236 hybrid_clock.cc:648] HybridClock initialized: now 1777601006427610 us; error 56 us; skew 500 ppm
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:26.430045 27236 init.cc:377] Logged in from keytab as oryx/127.18.96.194@KRBTEST.COM (short username oryx)
I20260501 02:03:26.430888 27236 webserver.cc:492] Webserver started at http://127.18.96.194:45623/ using document root <none> and password file <none>
I20260501 02:03:26.431348 27236 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:26.431411 27236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:26.431603 27236 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:26.433162 27236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "13c5181638b44f9691f1c216d58551a2"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "e8ecad8fde180cb61d502c5e3f157afe"
server_key_iv: "f8071a0f06c8a06e3fe3d307c0184537"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.433602 27236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "13c5181638b44f9691f1c216d58551a2"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "e8ecad8fde180cb61d502c5e3f157afe"
server_key_iv: "f8071a0f06c8a06e3fe3d307c0184537"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.436650 27236 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:26.438570 27251 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.439560 27236 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:26.439684 27236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "13c5181638b44f9691f1c216d58551a2"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "e8ecad8fde180cb61d502c5e3f157afe"
server_key_iv: "f8071a0f06c8a06e3fe3d307c0184537"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.439775 27236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:26.461834 27236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:26.464476 27236 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:26.464659 27236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:26.466060 27236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:26.466121 27236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.466199 27236 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:26.466236 27236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:26.476943 27236 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:34925
I20260501 02:03:26.477564 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.466647 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:49167 (local address 127.18.96.254:40755)
0501 02:03:26.466791 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.466797 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.467351 (+ 554us) server_negotiation.cc:408] Connection header received
0501 02:03:26.468122 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.468126 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.468183 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.468288 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.469628 (+ 1340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.470471 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.471446 (+ 975us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.471690 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.473976 (+ 2286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.473995 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.473999 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.474025 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.475699 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.476207 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.476211 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.476212 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.476266 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.476538 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.476542 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.476544 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.476781 (+ 237us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.476896 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.477362 (+ 466us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.477449 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:03:26.477939 27236 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:26.478129 27366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:34925 every 8 connection(s)
I20260501 02:03:26.478231 27260 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.466865 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.194:49167)
0501 02:03:26.467220 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.467253 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.467944 (+ 691us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.468440 (+ 496us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.468447 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.468830 (+ 383us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.469465 (+ 635us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.469481 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.470658 (+ 1177us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.470663 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.471323 (+ 660us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.471336 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.471566 (+ 230us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.472156 (+ 590us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.472181 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.473817 (+ 1636us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.475816 (+ 1999us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.475822 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.475831 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.476103 (+ 272us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.476370 (+ 267us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.476372 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.476373 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.476447 (+ 74us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.476907 (+ 460us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.476912 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.477133 (+ 221us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.477330 (+ 197us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.477527 (+ 197us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260501 02:03:26.479615 27367 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40755
I20260501 02:03:26.479768 27367 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:26.479748 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27236
I20260501 02:03:26.479884 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:26.480141 18819 external_mini_cluster.cc:1468] Setting key c2c687a5f432269c377a0674153f50d4
I20260501 02:03:26.480258 27367 heartbeater.cc:507] Master 127.18.96.254:40755 requested a full tablet report, sending...
I20260501 02:03:26.481122 27020 ts_manager.cc:194] Registered new tserver with Master: 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925)
I20260501 02:03:26.481791 27020 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:49167
I20260501 02:03:26.488011 27373 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.481167 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.194:34955)
0501 02:03:26.481431 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.481441 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.481550 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.481768 (+ 218us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.481771 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.481929 (+ 158us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.482200 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.482207 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.483387 (+ 1180us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.483392 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.483918 (+ 526us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.483926 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.484018 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.484483 (+ 465us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.484495 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.484744 (+ 249us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.486704 (+ 1960us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.486707 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.486708 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.486868 (+ 160us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.487171 (+ 303us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.487174 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.487175 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.487223 (+ 48us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.487649 (+ 426us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.487652 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.487717 (+ 65us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.487794 (+ 77us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.487890 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260501 02:03:26.488157 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.481206 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34955 (local address 127.18.96.254:40755)
0501 02:03:26.481310 (+ 104us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.481313 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.481505 (+ 192us) server_negotiation.cc:408] Connection header received
0501 02:03:26.481631 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.481634 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.481673 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.481755 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.482317 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.483012 (+ 695us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.484021 (+ 1009us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.484162 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.484866 (+ 704us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.484894 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.484900 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.484930 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.486595 (+ 1665us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.486967 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.486972 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.486977 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.487032 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.487323 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.487329 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.487333 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.487545 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.487647 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.487886 (+ 239us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.487989 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for oryx/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab.
I20260501 02:03:26.524957 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40755
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:26.620282 27378 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:26.620473 27378 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:26.620565 27378 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:26.623656 27378 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:26.623705 27378 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:26.623754 27378 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260501 02:03:26.623792 27378 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:26.627847 27378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/krb5kdc/oryx.keytab
--principal=oryx/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.96.254:40755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:26.628649 27378 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:26.629391 27378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:26.635159 27383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:26.635247 27386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:26.635275 27384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:26.635389 27378 server_base.cc:1061] running on GCE node
I20260501 02:03:26.635730 27378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:26.636227 27378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:26.637400 27378 hybrid_clock.cc:648] HybridClock initialized: now 1777601006637387 us; error 33 us; skew 500 ppm
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:26.639851 27378 init.cc:377] Logged in from keytab as oryx/127.18.96.195@KRBTEST.COM (short username oryx)
I20260501 02:03:26.640760 27378 webserver.cc:492] Webserver started at http://127.18.96.195:36855/ using document root <none> and password file <none>
I20260501 02:03:26.641208 27378 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:26.641250 27378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:26.641464 27378 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:26.642951 27378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "102caed678ea47eba2d0eac75d34e8e5"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "5de534a55d7c2be8e6170f9289cb465b"
server_key_iv: "e5c3d198be50dc038745f0e1c7593339"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.643380 27378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "102caed678ea47eba2d0eac75d34e8e5"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "5de534a55d7c2be8e6170f9289cb465b"
server_key_iv: "e5c3d198be50dc038745f0e1c7593339"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.646412 27378 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:03:26.648166 27393 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.648974 27378 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:26.649070 27378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "102caed678ea47eba2d0eac75d34e8e5"
format_stamp: "Formatted at 2026-05-01 02:03:26 on dist-test-slave-2j90"
server_key: "5de534a55d7c2be8e6170f9289cb465b"
server_key_iv: "e5c3d198be50dc038745f0e1c7593339"
server_key_version: "encryptionkey@0"
I20260501 02:03:26.649165 27378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:26.657300 27378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:26.659407 27378 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:26.659634 27378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:26.660916 27378 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:26.660970 27378 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:26.661011 27378 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:26.661039 27378 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:26.672210 27378 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:40119
I20260501 02:03:26.672330 27508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:40119 every 8 connection(s)
I20260501 02:03:26.672405 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.661370 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:43053 (local address 127.18.96.254:40755)
0501 02:03:26.661492 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.661495 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.662524 (+ 1029us) server_negotiation.cc:408] Connection header received
0501 02:03:26.663388 (+ 864us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.663392 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.663469 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.663562 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.664980 (+ 1418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.665574 (+ 594us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.666369 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.666562 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.668965 (+ 2403us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.668985 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.668988 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.669022 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.670581 (+ 1559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.671085 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.671088 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.671094 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.671134 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.671430 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.671434 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.671435 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.671598 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.671703 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.672147 (+ 444us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.672251 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:03:26.673044 27402 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.661634 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.195:43053)
0501 02:03:26.662406 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.662433 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.663211 (+ 778us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.663923 (+ 712us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.663935 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.664270 (+ 335us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.664844 (+ 574us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.664856 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.665744 (+ 888us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.665753 (+ 9us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.666258 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.666270 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.666448 (+ 178us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.666993 (+ 545us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.667010 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.668785 (+ 1775us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.670693 (+ 1908us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.670698 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.670708 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.670978 (+ 270us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.671216 (+ 238us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.671219 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.671221 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.671345 (+ 124us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.671702 (+ 357us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.671707 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.671892 (+ 185us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.672098 (+ 206us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.672314 (+ 216us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260501 02:03:26.673286 27378 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:26.674980 27509 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40755
I20260501 02:03:26.675158 27509 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:26.675565 27509 heartbeater.cc:507] Master 127.18.96.254:40755 requested a full tablet report, sending...
I20260501 02:03:26.676508 27020 ts_manager.cc:194] Registered new tserver with Master: 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:26.677280 27020 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:43053
I20260501 02:03:26.679872 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27378
I20260501 02:03:26.680022 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:26.680325 18819 external_mini_cluster.cc:1468] Setting key 77cf1e8f775601c2cc3d25b8a3e16c71
I20260501 02:03:26.684417 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:03:26.692024 27515 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.682540 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.18.96.195:58135)
0501 02:03:26.685007 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.685019 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.685132 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.685357 (+ 225us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.685361 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.685530 (+ 169us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:26.685730 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.685735 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.686517 (+ 782us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.686521 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.687182 (+ 661us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.687198 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.687704 (+ 506us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.688293 (+ 589us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:26.688317 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:26.688804 (+ 487us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:26.690618 (+ 1814us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.690621 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.690624 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.690781 (+ 157us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.691013 (+ 232us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:26.691015 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:26.691017 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:26.691056 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:26.691457 (+ 401us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:26.691462 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:26.691538 (+ 76us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.691654 (+ 116us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.691781 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2402,"thread_start_us":116,"threads_started":1}
I20260501 02:03:26.692770 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.682208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:58135 (local address 127.18.96.254:40755)
0501 02:03:26.682449 (+ 241us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.682453 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.685090 (+ 2637us) server_negotiation.cc:408] Connection header received
0501 02:03:26.685219 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.685222 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.685266 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.685343 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.685837 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.686359 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.687358 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.687568 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.688962 (+ 1394us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.688977 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.688980 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.689006 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.690501 (+ 1495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.690872 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.690876 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.690877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.690920 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.691135 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.691141 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.691145 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.691327 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.691415 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.692431 (+ 1016us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.692554 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260501 02:03:26.711988 27069 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:26.706539 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.18.96.254:40755)
0501 02:03:26.706655 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.706657 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.706666 (+ 9us) server_negotiation.cc:408] Connection header received
0501 02:03:26.706691 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.706693 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.706722 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.706815 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.707533 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.707973 (+ 440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.708652 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.708859 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.711842 (+ 2983us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38204: BlockingRecv error: recv got EOF from 127.0.0.1:38204 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 01 02:03:26 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601006, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:26.721597 27069 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.713654 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.18.96.254:40755)
0501 02:03:26.713752 (+ 98us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.713755 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.713906 (+ 151us) server_negotiation.cc:408] Connection header received
0501 02:03:26.714037 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.714039 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.714069 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.714127 (+ 58us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:26.715110 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.715540 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.716457 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.716581 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.718799 (+ 2218us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:26.718818 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:26.718820 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:26.718839 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:26.720129 (+ 1290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.720568 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.720572 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.720574 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.720619 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:26.720837 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:26.720842 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:26.720844 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:26.721007 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:26.721151 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.721357 (+ 206us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.721454 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:03:26.724139 27020 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:26.726555 27020 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:26.740731 27535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.734589 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:40119 (local address 127.0.0.1:36942)
0501 02:03:26.736123 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.736203 (+ 80us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.736326 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.737273 (+ 947us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.737278 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.737296 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:26.737520 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.737530 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739474 (+ 1944us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.739477 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.740285 (+ 808us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.740294 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.740352 (+ 58us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.740375 (+ 23us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.740451 (+ 76us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.740514 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":108,"threads_started":1}
I20260501 02:03:26.741479 27533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.734811 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34925 (local address 127.0.0.1:42174)
0501 02:03:26.736318 (+ 1507us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.736330 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.736410 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.737096 (+ 686us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.737104 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.737132 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:26.737452 (+ 320us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.737461 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739211 (+ 1750us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.739217 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.740325 (+ 1108us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.740336 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.740438 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.740460 (+ 22us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.740503 (+ 43us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.740570 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":106,"threads_started":1}
I20260501 02:03:26.741500 27536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.735971 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.18.96.193:45283)
0501 02:03:26.736573 (+ 602us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.736582 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.736631 (+ 49us) server_negotiation.cc:408] Connection header received
0501 02:03:26.736767 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.736772 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.736930 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.737062 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:26.737931 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739179 (+ 1248us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.740665 (+ 1486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.741152 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.741249 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.741309 (+ 60us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.741377 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":100,"threads_started":1}
I20260501 02:03:26.741729 27531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.735783 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.0.0.1:45580)
0501 02:03:26.736556 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.736568 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.736657 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.737226 (+ 569us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.737232 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.737256 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:26.737799 (+ 543us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.737805 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739311 (+ 1506us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.739314 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.740214 (+ 900us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.740222 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.740325 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.740375 (+ 50us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.740460 (+ 85us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.740514 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"spinlock_wait_cycles":768,"thread_start_us":132,"threads_started":1}
I20260501 02:03:26.742266 27532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.734753 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.18.96.195:40119)
0501 02:03:26.736897 (+ 2144us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.736902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.736922 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:03:26.736996 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.737003 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.737139 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.737406 (+ 267us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:26.737637 (+ 231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739364 (+ 1727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.740639 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.741153 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.741249 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.741309 (+ 60us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.741377 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2039,"thread_start_us":131,"threads_started":1}
I20260501 02:03:26.743949 27534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.735346 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.18.96.194:34925)
0501 02:03:26.735688 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.735696 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.736415 (+ 719us) server_negotiation.cc:408] Connection header received
0501 02:03:26.736525 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.736532 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.736729 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.736888 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:26.737598 (+ 710us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.739049 (+ 1451us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.742025 (+ 2976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.743375 (+ 1350us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.743513 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.743583 (+ 70us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.743741 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260501 02:03:26.744668 27443 tablet_service.cc:1511] Processing CreateTablet for tablet cb170071eab14457aa99dfc4f1b44c6b (DEFAULT_TABLE table=test-table [id=cc2d46d84d0947cabf7107f95af51c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.744668 27159 tablet_service.cc:1511] Processing CreateTablet for tablet cb170071eab14457aa99dfc4f1b44c6b (DEFAULT_TABLE table=test-table [id=cc2d46d84d0947cabf7107f95af51c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.745602 27159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb170071eab14457aa99dfc4f1b44c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.745602 27443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb170071eab14457aa99dfc4f1b44c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.746878 27301 tablet_service.cc:1511] Processing CreateTablet for tablet cb170071eab14457aa99dfc4f1b44c6b (DEFAULT_TABLE table=test-table [id=cc2d46d84d0947cabf7107f95af51c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.747856 27301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb170071eab14457aa99dfc4f1b44c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.753031 27538 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Bootstrap starting.
I20260501 02:03:26.754264 27537 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Bootstrap starting.
I20260501 02:03:26.755033 27538 tablet_bootstrap.cc:654] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.755509 27539 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Bootstrap starting.
I20260501 02:03:26.756333 27538 log.cc:826] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:26.757644 27537 tablet_bootstrap.cc:654] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.757872 27539 tablet_bootstrap.cc:654] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.758287 27538 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: No bootstrap required, opened a new log
I20260501 02:03:26.758428 27537 log.cc:826] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:26.758459 27538 ts_tablet_manager.cc:1403] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260501 02:03:26.759196 27539 log.cc:826] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:26.760897 27538 raft_consensus.cc:359] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.761062 27538 raft_consensus.cc:385] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.761093 27538 raft_consensus.cc:740] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.761145 27539 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: No bootstrap required, opened a new log
I20260501 02:03:26.761320 27539 ts_tablet_manager.cc:1403] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:03:26.761431 27538 consensus_queue.cc:260] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.761997 27225 heartbeater.cc:499] Master 127.18.96.254:40755 was elected leader, sending a full tablet report...
I20260501 02:03:26.762457 27538 ts_tablet_manager.cc:1434] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:03:26.764091 27537 tablet_bootstrap.cc:492] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: No bootstrap required, opened a new log
I20260501 02:03:26.764252 27537 ts_tablet_manager.cc:1403] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260501 02:03:26.765213 27539 raft_consensus.cc:359] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.765419 27539 raft_consensus.cc:385] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.765472 27539 raft_consensus.cc:740] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.765967 27539 consensus_queue.cc:260] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.766867 27539 ts_tablet_manager.cc:1434] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260501 02:03:26.767191 27367 heartbeater.cc:499] Master 127.18.96.254:40755 was elected leader, sending a full tablet report...
I20260501 02:03:26.767165 27537 raft_consensus.cc:359] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.767294 27537 raft_consensus.cc:385] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.767323 27537 raft_consensus.cc:740] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.767625 27537 consensus_queue.cc:260] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.768160 27509 heartbeater.cc:499] Master 127.18.96.254:40755 was elected leader, sending a full tablet report...
I20260501 02:03:26.768405 27537 ts_tablet_manager.cc:1434] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260501 02:03:26.770213 27226 tablet.cc:2404] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:26.830912 27545 raft_consensus.cc:493] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:26.831084 27545 raft_consensus.cc:515] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.832278 27545 leader_election.cc:290] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283)
I20260501 02:03:26.835343 27402 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.832465 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34925 (local address 127.18.96.195:35417)
0501 02:03:26.832689 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.832706 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.832797 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.833167 (+ 370us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.833175 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.833196 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:26.833450 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.833458 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.834366 (+ 908us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.834370 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.834964 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.834970 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.835090 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.835106 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.835158 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.835219 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260501 02:03:26.835846 27534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.832648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:35417 (local address 127.18.96.194:34925)
0501 02:03:26.832780 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.832784 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.832798 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:26.832931 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.832934 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.832985 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.833054 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:26.833576 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.834247 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.835077 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.835576 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.835610 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.835660 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.835726 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:03:26.836426 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb170071eab14457aa99dfc4f1b44c6b" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2" is_pre_election: true
I20260501 02:03:26.836556 27546 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.832547 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.18.96.195:41493)
0501 02:03:26.833012 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:26.833028 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:26.833136 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:26.833407 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:26.833412 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:26.833435 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:26.833802 (+ 367us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.833811 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.835021 (+ 1210us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.835028 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:26.836082 (+ 1054us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:26.836094 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.836220 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.836239 (+ 19us) client_negotiation.cc:770] Sending connection context
0501 02:03:26.836293 (+ 54us) client_negotiation.cc:241] Negotiation successful
0501 02:03:26.836351 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":132,"threads_started":1}
I20260501 02:03:26.836710 27321 raft_consensus.cc:2468] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 0.
I20260501 02:03:26.836995 27536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.832987 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:41493 (local address 127.18.96.193:45283)
0501 02:03:26.833110 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.833114 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.833129 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:26.833243 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.833246 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.833300 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.833377 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:26.833991 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.834865 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.836243 (+ 1378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.836700 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.836740 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.836802 (+ 62us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.836857 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:03:26.837198 27397 leader_election.cc:304] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102caed678ea47eba2d0eac75d34e8e5, 13c5181638b44f9691f1c216d58551a2; no voters:
I20260501 02:03:26.837585 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb170071eab14457aa99dfc4f1b44c6b" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9" is_pre_election: true
I20260501 02:03:26.837684 27545 raft_consensus.cc:2804] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:26.837742 27545 raft_consensus.cc:493] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:26.837783 27545 raft_consensus.cc:3060] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:26.837824 27179 raft_consensus.cc:2468] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 0.
I20260501 02:03:26.838868 27545 raft_consensus.cc:515] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.839239 27545 leader_election.cc:290] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 election: Requested vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283)
I20260501 02:03:26.839571 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb170071eab14457aa99dfc4f1b44c6b" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9"
I20260501 02:03:26.839598 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb170071eab14457aa99dfc4f1b44c6b" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2"
I20260501 02:03:26.839699 27179 raft_consensus.cc:3060] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:26.839717 27321 raft_consensus.cc:3060] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:26.840828 27321 raft_consensus.cc:2468] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 1.
I20260501 02:03:26.840828 27179 raft_consensus.cc:2468] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 1.
I20260501 02:03:26.841148 27394 leader_election.cc:304] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102caed678ea47eba2d0eac75d34e8e5, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:26.841313 27545 raft_consensus.cc:2804] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:26.841527 27545 raft_consensus.cc:697] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 1 LEADER]: Becoming Leader. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Running, Role: LEADER
I20260501 02:03:26.841809 27545 consensus_queue.cc:237] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } }
I20260501 02:03:26.844933 27019 catalog_manager.cc:5671] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195). New cstate: current_term: 1 leader_uuid: "102caed678ea47eba2d0eac75d34e8e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } health_report { overall_health: UNKNOWN } } }
I20260501 02:03:26.870003 27532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:26.867030 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.18.96.195:40119)
0501 02:03:26.867169 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:26.867176 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:26.867190 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:26.867365 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:26.867368 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:26.867417 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:26.867518 (+ 101us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:26.867872 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.868377 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:26.869017 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:26.869204 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:26.869266 (+ 62us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:26.869589 (+ 323us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:26.869704 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:26.869795 (+ 91us) server_negotiation.cc:300] Negotiation successful
0501 02:03:26.869838 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260501 02:03:26.874890 27179 raft_consensus.cc:1275] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Refusing update from remote peer 102caed678ea47eba2d0eac75d34e8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:26.874882 27321 raft_consensus.cc:1275] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Refusing update from remote peer 102caed678ea47eba2d0eac75d34e8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:26.875538 27545 consensus_queue.cc:1048] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:26.875954 27547 consensus_queue.cc:1048] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:26.884918 27551 mvcc.cc:204] Tried to move back new op lower bound from 7281053724153196544 to 7281053724026748928. Current Snapshot: MvccSnapshot[applied={T|T < 7281053724153196544}]
I20260501 02:03:26.885118 27552 mvcc.cc:204] Tried to move back new op lower bound from 7281053724153196544 to 7281053724026748928. Current Snapshot: MvccSnapshot[applied={T|T < 7281053724153196544}]
I20260501 02:03:26.892634 27020 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:26.892800 27020 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:26.895629 27020 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Sending DeleteTablet for 3 replicas of tablet cb170071eab14457aa99dfc4f1b44c6b
I20260501 02:03:26.896289 27443 tablet_service.cc:1558] Processing DeleteTablet for tablet cb170071eab14457aa99dfc4f1b44c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:26 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260501 02:03:26.896538 27159 tablet_service.cc:1558] Processing DeleteTablet for tablet cb170071eab14457aa99dfc4f1b44c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:26 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:45580
I20260501 02:03:26.897056 27561 tablet_replica.cc:333] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: stopping tablet replica
I20260501 02:03:26.897356 27561 raft_consensus.cc:2243] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:26.897405 27301 tablet_service.cc:1558] Processing DeleteTablet for tablet cb170071eab14457aa99dfc4f1b44c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:26 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42174
I20260501 02:03:26.897675 27561 raft_consensus.cc:2272] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:26.898728 27020 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:26.899201 27020 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:26.899207 27561 ts_tablet_manager.cc:1916] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:26.901903 27561 ts_tablet_manager.cc:1929] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:26.902048 27561 log.cc:1199] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/cb170071eab14457aa99dfc4f1b44c6b
I20260501 02:03:26.902377 27561 ts_tablet_manager.cc:1950] T cb170071eab14457aa99dfc4f1b44c6b P c578cc17dba44a8eb8115c3edd2878f9: Deleting consensus metadata
I20260501 02:03:26.903486 27004 catalog_manager.cc:5002] TS c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283): tablet cb170071eab14457aa99dfc4f1b44c6b (table test-table [id=cc2d46d84d0947cabf7107f95af51c2a]) successfully deleted
I20260501 02:03:26.905567 27159 tablet_service.cc:1511] Processing CreateTablet for tablet 6022c488388c49ad94752ad3a609494d (DEFAULT_TABLE table=test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.905715 27443 tablet_service.cc:1511] Processing CreateTablet for tablet 6022c488388c49ad94752ad3a609494d (DEFAULT_TABLE table=test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.905872 27159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6022c488388c49ad94752ad3a609494d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.906023 27443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6022c488388c49ad94752ad3a609494d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.908103 27537 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Bootstrap starting.
I20260501 02:03:26.908409 27538 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Bootstrap starting.
I20260501 02:03:26.909175 27537 tablet_bootstrap.cc:654] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.909644 27538 tablet_bootstrap.cc:654] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.909825 27301 tablet_service.cc:1511] Processing CreateTablet for tablet 6022c488388c49ad94752ad3a609494d (DEFAULT_TABLE table=test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:26.910145 27301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6022c488388c49ad94752ad3a609494d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:26.910874 27538 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: No bootstrap required, opened a new log
I20260501 02:03:26.910945 27538 ts_tablet_manager.cc:1403] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:26.911394 27538 raft_consensus.cc:359] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.911499 27538 raft_consensus.cc:385] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.911525 27538 raft_consensus.cc:740] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.911630 27538 consensus_queue.cc:260] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.911856 27538 ts_tablet_manager.cc:1434] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:26.912674 27539 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Bootstrap starting.
I20260501 02:03:26.913642 27539 tablet_bootstrap.cc:654] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:26.914758 27560 tablet_replica.cc:333] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: stopping tablet replica
I20260501 02:03:26.914983 27562 tablet_replica.cc:333] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: stopping tablet replica
I20260501 02:03:26.915253 27539 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: No bootstrap required, opened a new log
I20260501 02:03:26.915338 27539 ts_tablet_manager.cc:1403] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:26.915372 27562 raft_consensus.cc:2243] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:26.915792 27539 raft_consensus.cc:359] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.915891 27539 raft_consensus.cc:385] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.915930 27539 raft_consensus.cc:740] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.916040 27539 consensus_queue.cc:260] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.916292 27539 ts_tablet_manager.cc:1434] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:26.915709 27562 raft_consensus.cc:2272] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:26.916968 27560 raft_consensus.cc:2243] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:26.917363 27560 raft_consensus.cc:2272] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:26.917536 27537 tablet_bootstrap.cc:492] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: No bootstrap required, opened a new log
I20260501 02:03:26.917634 27537 ts_tablet_manager.cc:1403] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260501 02:03:26.918188 27537 raft_consensus.cc:359] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.918318 27537 raft_consensus.cc:385] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:26.918362 27537 raft_consensus.cc:740] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Initialized, Role: FOLLOWER
I20260501 02:03:26.918490 27537 consensus_queue.cc:260] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:26.918768 27537 ts_tablet_manager.cc:1434] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260501 02:03:26.918849 27562 ts_tablet_manager.cc:1916] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:26.921581 27562 ts_tablet_manager.cc:1929] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:26.921700 27562 log.cc:1199] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/cb170071eab14457aa99dfc4f1b44c6b
I20260501 02:03:26.922055 27562 ts_tablet_manager.cc:1950] T cb170071eab14457aa99dfc4f1b44c6b P 13c5181638b44f9691f1c216d58551a2: Deleting consensus metadata
I20260501 02:03:26.923220 27007 catalog_manager.cc:5002] TS 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925): tablet cb170071eab14457aa99dfc4f1b44c6b (table test-table [id=cc2d46d84d0947cabf7107f95af51c2a]) successfully deleted
W20260501 02:03:26.924283 27510 tablet.cc:2404] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:26.924293 27560 ts_tablet_manager.cc:1916] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:26.927230 27560 ts_tablet_manager.cc:1929] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:26.927350 27560 log.cc:1199] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/cb170071eab14457aa99dfc4f1b44c6b
I20260501 02:03:26.927680 27560 ts_tablet_manager.cc:1950] T cb170071eab14457aa99dfc4f1b44c6b P 102caed678ea47eba2d0eac75d34e8e5: Deleting consensus metadata
I20260501 02:03:26.928751 27005 catalog_manager.cc:5002] TS 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119): tablet cb170071eab14457aa99dfc4f1b44c6b (table test-table [id=cc2d46d84d0947cabf7107f95af51c2a]) successfully deleted
W20260501 02:03:26.979578 27368 tablet.cc:2404] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:27.150274 27543 raft_consensus.cc:493] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:27.150406 27543 raft_consensus.cc:515] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.151645 27543 leader_election.cc:290] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.154983 27566 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.151788 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34925 (local address 127.18.96.193:42355)
0501 02:03:27.152082 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.152097 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.152252 (+ 155us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.152644 (+ 392us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.152650 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.152673 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:27.152944 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.152951 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.153880 (+ 929us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.153884 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.154672 (+ 788us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.154680 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.154769 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.154782 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.154819 (+ 37us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.154858 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260501 02:03:27.155499 27534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.151930 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:42355 (local address 127.18.96.194:34925)
0501 02:03:27.152080 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.152085 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.152169 (+ 84us) server_negotiation.cc:408] Connection header received
0501 02:03:27.152442 (+ 273us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.152446 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.152515 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.152590 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:27.153061 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.153777 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.154793 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.155264 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.155298 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.155342 (+ 44us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.155389 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:03:27.155975 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6022c488388c49ad94752ad3a609494d" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2" is_pre_election: true
I20260501 02:03:27.156123 27567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.151931 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:40119 (local address 127.18.96.193:37547)
0501 02:03:27.152216 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.152227 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.152330 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.152697 (+ 367us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.152700 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.152721 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:27.153419 (+ 698us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.153428 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.154354 (+ 926us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.154356 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.155100 (+ 744us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.155108 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.155765 (+ 657us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.155776 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.155904 (+ 128us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.155964 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260501 02:03:27.156148 27532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.152126 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:37547 (local address 127.18.96.195:40119)
0501 02:03:27.152260 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.152264 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.152279 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:27.152513 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.152517 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.152581 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.152674 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:27.153552 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.154251 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.155239 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.155688 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.155715 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.155934 (+ 219us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.156012 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:03:27.156139 27321 raft_consensus.cc:2468] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 0.
I20260501 02:03:27.156483 27113 leader_election.cc:304] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c5181638b44f9691f1c216d58551a2, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.156603 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6022c488388c49ad94752ad3a609494d" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5" is_pre_election: true
I20260501 02:03:27.156751 27543 raft_consensus.cc:2804] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:27.156844 27543 raft_consensus.cc:493] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:27.156848 27459 raft_consensus.cc:2468] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 0.
I20260501 02:03:27.156914 27543 raft_consensus.cc:3060] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.157809 27543 raft_consensus.cc:515] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.158174 27543 leader_election.cc:290] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 election: Requested vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.158469 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6022c488388c49ad94752ad3a609494d" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2"
I20260501 02:03:27.158476 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6022c488388c49ad94752ad3a609494d" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5"
I20260501 02:03:27.158593 27321 raft_consensus.cc:3060] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.158608 27459 raft_consensus.cc:3060] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.159441 27321 raft_consensus.cc:2468] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 1.
I20260501 02:03:27.159564 27459 raft_consensus.cc:2468] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 1.
I20260501 02:03:27.159722 27113 leader_election.cc:304] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c5181638b44f9691f1c216d58551a2, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.159900 27543 raft_consensus.cc:2804] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:27.160077 27543 raft_consensus.cc:697] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 1 LEADER]: Becoming Leader. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Running, Role: LEADER
I20260501 02:03:27.160283 27543 consensus_queue.cc:237] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.162304 27020 catalog_manager.cc:5671] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } health_report { overall_health: UNKNOWN } } }
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601005, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.254@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:27.234793 27576 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.227474 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40755 (local address 127.0.0.1:38222)
0501 02:03:27.227798 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.227811 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.227907 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.228143 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.228146 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.228541 (+ 395us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:27.228734 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.228740 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.229408 (+ 668us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.229410 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.229771 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.229776 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.229857 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.230390 (+ 533us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:27.230411 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:27.231818 (+ 1407us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:27.233435 (+ 1617us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:27.233441 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:27.233445 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:27.233634 (+ 189us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:27.233833 (+ 199us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:27.233835 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:27.233836 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:27.233876 (+ 40us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:27.234281 (+ 405us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:27.234285 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:27.234368 (+ 83us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.234486 (+ 118us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.234642 (+ 156us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260501 02:03:27.234905 27575 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.227507 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.18.96.254:40755)
0501 02:03:27.227757 (+ 250us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.227761 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.227874 (+ 113us) server_negotiation.cc:408] Connection header received
0501 02:03:27.227987 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.227990 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.228030 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.228111 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:27.228828 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.229324 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.229854 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.230040 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.231938 (+ 1898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:27.231952 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:27.231954 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:27.231975 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:27.233340 (+ 1365us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:27.233722 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:27.233725 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:27.233726 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:27.233763 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:27.234041 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:27.234045 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:27.234047 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:27.234183 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:27.234278 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.234538 (+ 260us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.234728 (+ 190us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260501 02:03:27.238528 27020 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38222:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260501 02:03:27.239259 27020 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:27.244575 27159 tablet_service.cc:1511] Processing CreateTablet for tablet 20f1e9bacae14159a1d2c1807571b08a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df8ee767660843069d5d7aa870df57a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:03:27.245023 27159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f1e9bacae14159a1d2c1807571b08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.244940 27301 tablet_service.cc:1511] Processing CreateTablet for tablet 20f1e9bacae14159a1d2c1807571b08a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df8ee767660843069d5d7aa870df57a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:03:27.245093 27443 tablet_service.cc:1511] Processing CreateTablet for tablet 20f1e9bacae14159a1d2c1807571b08a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df8ee767660843069d5d7aa870df57a8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260501 02:03:27.245301 27301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f1e9bacae14159a1d2c1807571b08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.245468 27443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f1e9bacae14159a1d2c1807571b08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.247782 27537 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: Bootstrap starting.
I20260501 02:03:27.247943 27539 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: Bootstrap starting.
I20260501 02:03:27.247943 27538 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: Bootstrap starting.
I20260501 02:03:27.248560 27537 tablet_bootstrap.cc:654] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.248982 27538 tablet_bootstrap.cc:654] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.248982 27539 tablet_bootstrap.cc:654] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.250543 27539 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: No bootstrap required, opened a new log
I20260501 02:03:27.250561 27537 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: No bootstrap required, opened a new log
I20260501 02:03:27.250643 27539 ts_tablet_manager.cc:1403] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:27.250653 27537 ts_tablet_manager.cc:1403] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260501 02:03:27.250895 27538 tablet_bootstrap.cc:492] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: No bootstrap required, opened a new log
I20260501 02:03:27.250988 27538 ts_tablet_manager.cc:1403] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:27.251133 27539 raft_consensus.cc:359] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251233 27539 raft_consensus.cc:385] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.251176 27537 raft_consensus.cc:359] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251268 27539 raft_consensus.cc:740] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.251274 27537 raft_consensus.cc:385] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.251307 27537 raft_consensus.cc:740] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.251395 27539 consensus_queue.cc:260] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251437 27537 consensus_queue.cc:260] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251470 27538 raft_consensus.cc:359] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251557 27538 raft_consensus.cc:385] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.251601 27538 raft_consensus.cc:740] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.251668 27539 ts_tablet_manager.cc:1434] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:03:27.251722 27537 ts_tablet_manager.cc:1434] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260501 02:03:27.251714 27538 consensus_queue.cc:260] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.251924 27543 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.251956 27556 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.252027 27543 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.252041 27556 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.251627 27544 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.252187 27544 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.252648 27538 ts_tablet_manager.cc:1434] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:27.337605 27543 raft_consensus.cc:493] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:27.337746 27543 raft_consensus.cc:515] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.338225 27543 leader_election.cc:290] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.338644 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f1e9bacae14159a1d2c1807571b08a" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2" is_pre_election: true
I20260501 02:03:27.338682 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f1e9bacae14159a1d2c1807571b08a" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5" is_pre_election: true
I20260501 02:03:27.338793 27459 raft_consensus.cc:2468] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 0.
I20260501 02:03:27.338793 27321 raft_consensus.cc:2468] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 0.
I20260501 02:03:27.339089 27113 leader_election.cc:304] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c5181638b44f9691f1c216d58551a2, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.339272 27543 raft_consensus.cc:2804] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:27.339350 27543 raft_consensus.cc:493] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:27.339402 27543 raft_consensus.cc:3060] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.340296 27543 raft_consensus.cc:515] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.340642 27543 leader_election.cc:290] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 election: Requested vote from peers 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.340957 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f1e9bacae14159a1d2c1807571b08a" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2"
I20260501 02:03:27.340997 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f1e9bacae14159a1d2c1807571b08a" candidate_uuid: "c578cc17dba44a8eb8115c3edd2878f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5"
I20260501 02:03:27.341073 27321 raft_consensus.cc:3060] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.341125 27459 raft_consensus.cc:3060] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.341853 27321 raft_consensus.cc:2468] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 1.
I20260501 02:03:27.341940 27459 raft_consensus.cc:2468] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c578cc17dba44a8eb8115c3edd2878f9 in term 1.
I20260501 02:03:27.342238 27113 leader_election.cc:304] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c5181638b44f9691f1c216d58551a2, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.342407 27543 raft_consensus.cc:2804] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:27.342515 27543 raft_consensus.cc:697] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [term 1 LEADER]: Becoming Leader. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Running, Role: LEADER
I20260501 02:03:27.342653 27543 consensus_queue.cc:237] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.343232 27568 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: TxnStatusTablet state changed. Reason: New leader c578cc17dba44a8eb8115c3edd2878f9. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.343335 27568 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:03:27.343654 27581 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260501 02:03:27.343787 27581 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260501 02:03:27.344245 27018 catalog_manager.cc:5671] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20260501 02:03:27.371104 27584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.367913 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.18.96.193:45283)
0501 02:03:27.368176 (+ 263us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.368179 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.368193 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:27.368255 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.368257 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.368302 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.368388 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.368771 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.369306 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.370236 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.370386 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.370426 (+ 40us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.370728 (+ 302us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.370804 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.370936 (+ 132us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.370974 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
I20260501 02:03:27.371117 27576 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.367795 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.0.0.1:45584)
0501 02:03:27.368023 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.368039 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.368146 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.368415 (+ 269us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.368418 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.368426 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:03:27.368660 (+ 234us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.368666 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.369425 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.369429 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.370128 (+ 699us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.370137 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.370260 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.370298 (+ 38us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.370818 (+ 520us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.370828 (+ 10us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.370908 (+ 80us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.370970 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260501 02:03:27.373545 27321 raft_consensus.cc:1275] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Refusing update from remote peer c578cc17dba44a8eb8115c3edd2878f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.373646 27459 raft_consensus.cc:1275] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Refusing update from remote peer c578cc17dba44a8eb8115c3edd2878f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.374019 27568 consensus_queue.cc:1048] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.374150 27543 consensus_queue.cc:1048] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.375770 27544 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: TxnStatusTablet state changed. Reason: New leader c578cc17dba44a8eb8115c3edd2878f9. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.375878 27544 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.376453 27556 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: TxnStatusTablet state changed. Reason: New leader c578cc17dba44a8eb8115c3edd2878f9. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.376549 27556 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.377756 27568 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.377846 27568 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:03:27.377803 27543 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.377884 27543 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P c578cc17dba44a8eb8115c3edd2878f9: This TxnStatusTablet replica's current role is: LEADER
I20260501 02:03:27.378232 27550 mvcc.cc:204] Tried to move back new op lower bound from 7281053726197587968 to 7281053726076665856. Current Snapshot: MvccSnapshot[applied={T|T < 7281053726197587968}]
I20260501 02:03:27.378371 27556 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.378455 27556 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 102caed678ea47eba2d0eac75d34e8e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.378732 27544 tablet_replica.cc:442] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c578cc17dba44a8eb8115c3edd2878f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } } }
I20260501 02:03:27.378818 27544 tablet_replica.cc:445] T 20f1e9bacae14159a1d2c1807571b08a P 13c5181638b44f9691f1c216d58551a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260501 02:03:27.385560 27584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.382598 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.18.96.193:45283)
0501 02:03:27.382800 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.382803 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.382894 (+ 91us) server_negotiation.cc:408] Connection header received
0501 02:03:27.383032 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.383034 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.383072 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.383133 (+ 61us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.383573 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.384071 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.384927 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.385118 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.385157 (+ 39us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.385219 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.385278 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.385396 (+ 118us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.385431 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260501 02:03:27.393968 27584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.391455 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57537 (local address 127.18.96.193:45283)
0501 02:03:27.391575 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.391579 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.391731 (+ 152us) server_negotiation.cc:408] Connection header received
0501 02:03:27.391857 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.391860 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.391899 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.391972 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.392289 (+ 317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.392690 (+ 401us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.393327 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.393492 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.393522 (+ 30us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.393574 (+ 52us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.393650 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.393771 (+ 121us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.393803 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260501 02:03:27.393962 27592 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.391411 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.18.96.193:57537)
0501 02:03:27.391662 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.391672 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.391772 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.391979 (+ 207us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.391982 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.391989 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:03:27.392195 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.392199 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.392826 (+ 627us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.392833 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.393227 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.393232 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.393345 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.393378 (+ 33us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.393669 (+ 291us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.393677 (+ 8us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.393745 (+ 68us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.393812 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260501 02:03:27.400036 27321 raft_consensus.cc:1275] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Refusing update from remote peer c578cc17dba44a8eb8115c3edd2878f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.400058 27459 raft_consensus.cc:1275] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Refusing update from remote peer c578cc17dba44a8eb8115c3edd2878f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.400409 27568 consensus_queue.cc:1048] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.400844 27587 consensus_queue.cc:1048] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.438792 27018 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.438944 27018 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.440781 27018 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Sending DeleteTablet for 3 replicas of tablet 6022c488388c49ad94752ad3a609494d
I20260501 02:03:27.441278 27159 tablet_service.cc:1558] Processing DeleteTablet for tablet 6022c488388c49ad94752ad3a609494d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:45580
I20260501 02:03:27.441337 27301 tablet_service.cc:1558] Processing DeleteTablet for tablet 6022c488388c49ad94752ad3a609494d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42174
I20260501 02:03:27.441444 27443 tablet_service.cc:1558] Processing DeleteTablet for tablet 6022c488388c49ad94752ad3a609494d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260501 02:03:27.441599 27597 tablet_replica.cc:333] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: stopping tablet replica
I20260501 02:03:27.441705 27597 raft_consensus.cc:2243] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:27.441804 27597 raft_consensus.cc:2272] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.442051 27598 tablet_replica.cc:333] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: stopping tablet replica
I20260501 02:03:27.442155 27598 raft_consensus.cc:2243] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:27.442539 27598 raft_consensus.cc:2272] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.442647 27597 ts_tablet_manager.cc:1916] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.443226 27598 ts_tablet_manager.cc:1916] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.445677 27598 ts_tablet_manager.cc:1929] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:03:27.445763 27598 log.cc:1199] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/6022c488388c49ad94752ad3a609494d
I20260501 02:03:27.441808 27599 tablet_replica.cc:333] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: stopping tablet replica
I20260501 02:03:27.446040 27598 ts_tablet_manager.cc:1950] T 6022c488388c49ad94752ad3a609494d P c578cc17dba44a8eb8115c3edd2878f9: Deleting consensus metadata
I20260501 02:03:27.446024 27599 raft_consensus.cc:2243] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:27.446262 27599 raft_consensus.cc:2272] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.446512 27597 ts_tablet_manager.cc:1929] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:03:27.446617 27597 log.cc:1199] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/6022c488388c49ad94752ad3a609494d
I20260501 02:03:27.446612 27004 catalog_manager.cc:5002] TS c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283): tablet 6022c488388c49ad94752ad3a609494d (table test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]) successfully deleted
I20260501 02:03:27.446902 27599 ts_tablet_manager.cc:1916] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.446933 27597 ts_tablet_manager.cc:1950] T 6022c488388c49ad94752ad3a609494d P 13c5181638b44f9691f1c216d58551a2: Deleting consensus metadata
I20260501 02:03:27.447615 27007 catalog_manager.cc:5002] TS 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925): tablet 6022c488388c49ad94752ad3a609494d (table test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]) successfully deleted
I20260501 02:03:27.449488 27599 ts_tablet_manager.cc:1929] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:03:27.449572 27599 log.cc:1199] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/6022c488388c49ad94752ad3a609494d
I20260501 02:03:27.449882 27599 ts_tablet_manager.cc:1950] T 6022c488388c49ad94752ad3a609494d P 102caed678ea47eba2d0eac75d34e8e5: Deleting consensus metadata
I20260501 02:03:27.450491 27005 catalog_manager.cc:5002] TS 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119): tablet 6022c488388c49ad94752ad3a609494d (table test-table [id=5b0ef9ee9a3b4ca4b05c38a262864215]) successfully deleted
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[26972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260501 02:03:27.469722 27575 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:27.463641 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.18.96.254:40755)
0501 02:03:27.463795 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.463799 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.463976 (+ 177us) server_negotiation.cc:408] Connection header received
0501 02:03:27.464132 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.464135 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.464181 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.464252 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:27.465274 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.465783 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.466637 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.466788 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.469601 (+ 2813us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38238: BlockingRecv error: recv got EOF from 127.0.0.1:38238 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[26972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM
I20260501 02:03:27.478457 27575 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.471114 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.18.96.254:40755)
0501 02:03:27.471221 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.471224 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.471410 (+ 186us) server_negotiation.cc:408] Connection header received
0501 02:03:27.471492 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.471496 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.471542 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.471615 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:27.472251 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.472968 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.473490 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.473669 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.475493 (+ 1824us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:27.475509 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:27.475512 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:27.475539 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:27.476949 (+ 1410us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:27.477394 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:27.477398 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:27.477400 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:27.477449 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:27.477681 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:27.477684 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:27.477686 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:27.477835 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:27.478006 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.478195 (+ 189us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.478314 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:03:27.480906 27018 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:27.481365 27018 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:27.486192 27159 tablet_service.cc:1511] Processing CreateTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (DEFAULT_TABLE table=test-table [id=8b99487f563842e5a03d61df34384cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.486466 27159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.488711 27538 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Bootstrap starting.
I20260501 02:03:27.489018 27301 tablet_service.cc:1511] Processing CreateTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (DEFAULT_TABLE table=test-table [id=8b99487f563842e5a03d61df34384cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.489284 27301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.489336 27443 tablet_service.cc:1511] Processing CreateTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (DEFAULT_TABLE table=test-table [id=8b99487f563842e5a03d61df34384cd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.489512 27538 tablet_bootstrap.cc:654] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.489627 27443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.491153 27539 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Bootstrap starting.
I20260501 02:03:27.492110 27539 tablet_bootstrap.cc:654] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.492877 27537 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Bootstrap starting.
I20260501 02:03:27.493781 27537 tablet_bootstrap.cc:654] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.494498 27538 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: No bootstrap required, opened a new log
I20260501 02:03:27.494592 27538 ts_tablet_manager.cc:1403] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260501 02:03:27.494881 27539 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: No bootstrap required, opened a new log
I20260501 02:03:27.494974 27539 ts_tablet_manager.cc:1403] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260501 02:03:27.495038 27538 raft_consensus.cc:359] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.495131 27538 raft_consensus.cc:385] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.495158 27538 raft_consensus.cc:740] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.495268 27538 consensus_queue.cc:260] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.495355 27539 raft_consensus.cc:359] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.495455 27539 raft_consensus.cc:385] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.495483 27539 raft_consensus.cc:740] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.495528 27538 ts_tablet_manager.cc:1434] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:03:27.495595 27539 consensus_queue.cc:260] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.495843 27539 ts_tablet_manager.cc:1434] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:27.498078 27537 tablet_bootstrap.cc:492] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: No bootstrap required, opened a new log
I20260501 02:03:27.498167 27537 ts_tablet_manager.cc:1403] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260501 02:03:27.498559 27537 raft_consensus.cc:359] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.498674 27537 raft_consensus.cc:385] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.498713 27537 raft_consensus.cc:740] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.498871 27537 consensus_queue.cc:260] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.499133 27537 ts_tablet_manager.cc:1434] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:27.509454 27556 raft_consensus.cc:493] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:27.509545 27556 raft_consensus.cc:515] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.509945 27556 leader_election.cc:290] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283), 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925)
I20260501 02:03:27.510267 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e15cf0fc8434d9ab8aaf2a12252e5a7" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2" is_pre_election: true
I20260501 02:03:27.510280 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e15cf0fc8434d9ab8aaf2a12252e5a7" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9" is_pre_election: true
I20260501 02:03:27.510390 27321 raft_consensus.cc:2468] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 0.
I20260501 02:03:27.510398 27179 raft_consensus.cc:2468] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 0.
I20260501 02:03:27.510679 27397 leader_election.cc:304] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102caed678ea47eba2d0eac75d34e8e5, 13c5181638b44f9691f1c216d58551a2; no voters:
I20260501 02:03:27.510838 27556 raft_consensus.cc:2804] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:27.510898 27556 raft_consensus.cc:493] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:27.510960 27556 raft_consensus.cc:3060] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.511708 27556 raft_consensus.cc:515] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.512039 27556 leader_election.cc:290] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 election: Requested vote from peers c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283), 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925)
I20260501 02:03:27.512302 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e15cf0fc8434d9ab8aaf2a12252e5a7" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9"
I20260501 02:03:27.512408 27321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e15cf0fc8434d9ab8aaf2a12252e5a7" candidate_uuid: "102caed678ea47eba2d0eac75d34e8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c5181638b44f9691f1c216d58551a2"
I20260501 02:03:27.512434 27179 raft_consensus.cc:3060] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.512501 27321 raft_consensus.cc:3060] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.513404 27179 raft_consensus.cc:2468] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 1.
I20260501 02:03:27.513552 27321 raft_consensus.cc:2468] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 102caed678ea47eba2d0eac75d34e8e5 in term 1.
I20260501 02:03:27.513701 27394 leader_election.cc:304] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102caed678ea47eba2d0eac75d34e8e5, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.514000 27556 raft_consensus.cc:2804] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:27.514173 27556 raft_consensus.cc:697] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 1 LEADER]: Becoming Leader. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Running, Role: LEADER
I20260501 02:03:27.514305 27556 consensus_queue.cc:237] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.515689 27018 catalog_manager.cc:5671] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195). New cstate: current_term: 1 leader_uuid: "102caed678ea47eba2d0eac75d34e8e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } health_report { overall_health: HEALTHY } } }
I20260501 02:03:27.524072 27532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.521682 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.18.96.195:40119)
0501 02:03:27.521810 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.521813 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.521823 (+ 10us) server_negotiation.cc:408] Connection header received
0501 02:03:27.521854 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.521857 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.521893 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.522005 (+ 112us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.522333 (+ 328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.522796 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.523423 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.523604 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.523639 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.523702 (+ 63us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.523799 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.523918 (+ 119us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.523955 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:03:27.525678 27179 raft_consensus.cc:1275] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Refusing update from remote peer 102caed678ea47eba2d0eac75d34e8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.525694 27321 raft_consensus.cc:1275] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Refusing update from remote peer 102caed678ea47eba2d0eac75d34e8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.526111 27556 consensus_queue.cc:1048] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.526281 27615 consensus_queue.cc:1048] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.534433 27018 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.534559 27018 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.536347 27018 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Sending DeleteTablet for 3 replicas of tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7
I20260501 02:03:27.536801 27159 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:45580
I20260501 02:03:27.536835 27301 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42174
I20260501 02:03:27.536996 27597 tablet_replica.cc:333] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: stopping tablet replica
I20260501 02:03:27.536989 27598 tablet_replica.cc:333] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: stopping tablet replica
I20260501 02:03:27.537107 27597 raft_consensus.cc:2243] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:27.537122 27598 raft_consensus.cc:2243] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:27.537267 27598 raft_consensus.cc:2272] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.537262 27597 raft_consensus.cc:2272] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.537480 27019 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260501 02:03:27.537577 27443 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260501 02:03:27.537766 27599 tablet_replica.cc:333] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: stopping tablet replica
I20260501 02:03:27.537883 27599 raft_consensus.cc:2243] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 1 LEADER]: Raft consensus shutting down.
W20260501 02:03:27.537870 27019 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:27.538085 27599 raft_consensus.cc:2272] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.538319 27597 ts_tablet_manager.cc:1916] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.538797 27599 ts_tablet_manager.cc:1916] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.540437 27597 ts_tablet_manager.cc:1929] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:27.540513 27597 log.cc:1199] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/2e15cf0fc8434d9ab8aaf2a12252e5a7
I20260501 02:03:27.540766 27597 ts_tablet_manager.cc:1950] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 13c5181638b44f9691f1c216d58551a2: Deleting consensus metadata
I20260501 02:03:27.540884 27599 ts_tablet_manager.cc:1929] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:27.540961 27599 log.cc:1199] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/2e15cf0fc8434d9ab8aaf2a12252e5a7
I20260501 02:03:27.541188 27599 ts_tablet_manager.cc:1950] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P 102caed678ea47eba2d0eac75d34e8e5: Deleting consensus metadata
I20260501 02:03:27.541543 27007 catalog_manager.cc:5002] TS 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925): tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (table test-table [id=8b99487f563842e5a03d61df34384cd9]) successfully deleted
I20260501 02:03:27.541700 27598 ts_tablet_manager.cc:1916] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.541823 27005 catalog_manager.cc:5002] TS 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119): tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (table test-table [id=8b99487f563842e5a03d61df34384cd9]) successfully deleted
I20260501 02:03:27.543815 27598 ts_tablet_manager.cc:1929] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:27.543883 27598 log.cc:1199] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/2e15cf0fc8434d9ab8aaf2a12252e5a7
I20260501 02:03:27.544137 27598 ts_tablet_manager.cc:1950] T 2e15cf0fc8434d9ab8aaf2a12252e5a7 P c578cc17dba44a8eb8115c3edd2878f9: Deleting consensus metadata
I20260501 02:03:27.544780 27004 catalog_manager.cc:5002] TS c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283): tablet 2e15cf0fc8434d9ab8aaf2a12252e5a7 (table test-table [id=8b99487f563842e5a03d61df34384cd9]) successfully deleted
I20260501 02:03:27.545636 27159 tablet_service.cc:1511] Processing CreateTablet for tablet 1a94778c66a745a38b360eaba2dd52ec (DEFAULT_TABLE table=test-table [id=4fd62acfad6b4ff78ceaa5916a6a0963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.545889 27159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a94778c66a745a38b360eaba2dd52ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.546083 27301 tablet_service.cc:1511] Processing CreateTablet for tablet 1a94778c66a745a38b360eaba2dd52ec (DEFAULT_TABLE table=test-table [id=4fd62acfad6b4ff78ceaa5916a6a0963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.546320 27301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a94778c66a745a38b360eaba2dd52ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.547864 27538 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9: Bootstrap starting.
I20260501 02:03:27.548308 27539 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Bootstrap starting.
I20260501 02:03:27.548694 27538 tablet_bootstrap.cc:654] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.549288 27539 tablet_bootstrap.cc:654] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.549953 27538 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9: No bootstrap required, opened a new log
I20260501 02:03:27.550081 27538 ts_tablet_manager.cc:1403] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260501 02:03:27.550290 27443 tablet_service.cc:1511] Processing CreateTablet for tablet 1a94778c66a745a38b360eaba2dd52ec (DEFAULT_TABLE table=test-table [id=4fd62acfad6b4ff78ceaa5916a6a0963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:27.550525 27443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a94778c66a745a38b360eaba2dd52ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:27.550485 27538 raft_consensus.cc:359] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.550580 27538 raft_consensus.cc:385] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.550611 27538 raft_consensus.cc:740] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c578cc17dba44a8eb8115c3edd2878f9, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.550717 27538 consensus_queue.cc:260] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.550998 27538 ts_tablet_manager.cc:1434] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:03:27.552330 27537 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Bootstrap starting.
I20260501 02:03:27.553032 27539 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: No bootstrap required, opened a new log
I20260501 02:03:27.553105 27539 ts_tablet_manager.cc:1403] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260501 02:03:27.553267 27537 tablet_bootstrap.cc:654] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:27.553422 27539 raft_consensus.cc:359] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.553489 27539 raft_consensus.cc:385] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.553508 27539 raft_consensus.cc:740] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.553627 27539 consensus_queue.cc:260] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.553891 27539 ts_tablet_manager.cc:1434] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:03:27.555239 27537 tablet_bootstrap.cc:492] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: No bootstrap required, opened a new log
I20260501 02:03:27.555313 27537 ts_tablet_manager.cc:1403] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260501 02:03:27.555650 27537 raft_consensus.cc:359] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.555747 27537 raft_consensus.cc:385] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:27.555771 27537 raft_consensus.cc:740] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102caed678ea47eba2d0eac75d34e8e5, State: Initialized, Role: FOLLOWER
I20260501 02:03:27.555873 27537 consensus_queue.cc:260] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.556089 27537 ts_tablet_manager.cc:1434] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260501 02:03:27.665597 27544 raft_consensus.cc:493] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:27.665746 27544 raft_consensus.cc:515] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.667065 27544 leader_election.cc:290] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.670486 27625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.667249 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:40119 (local address 127.18.96.194:46297)
0501 02:03:27.667787 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.667799 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.667919 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.668485 (+ 566us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.668491 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.668513 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:27.668710 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.668715 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.669386 (+ 671us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.669389 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.669998 (+ 609us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.670024 (+ 26us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.670153 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.670223 (+ 70us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.670277 (+ 54us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.670329 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":225,"spinlock_wait_cycles":18048,"thread_start_us":43,"threads_started":1}
I20260501 02:03:27.670995 27532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.667448 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46297 (local address 127.18.96.195:40119)
0501 02:03:27.667592 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.667598 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.667928 (+ 330us) server_negotiation.cc:408] Connection header received
0501 02:03:27.668330 (+ 402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.668334 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.668383 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.668459 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:27.668822 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.669277 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.670134 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.670599 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.670638 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.670724 (+ 86us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.670848 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:03:27.671366 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a94778c66a745a38b360eaba2dd52ec" candidate_uuid: "13c5181638b44f9691f1c216d58551a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5" is_pre_election: true
I20260501 02:03:27.671518 27459 raft_consensus.cc:2468] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c5181638b44f9691f1c216d58551a2 in term 0.
I20260501 02:03:27.671748 27584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.667445 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:45747 (local address 127.18.96.193:45283)
0501 02:03:27.667590 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.667594 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.667883 (+ 289us) server_negotiation.cc:408] Connection header received
0501 02:03:27.668024 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.668028 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.668082 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.668174 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:27.668599 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.669332 (+ 733us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.670830 (+ 1498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.671378 (+ 548us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.671411 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.671574 (+ 163us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.671634 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:03:27.671785 27624 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.667249 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.18.96.194:45747)
0501 02:03:27.667776 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.667792 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.667929 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.668191 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.668193 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.668217 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:27.668421 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.668426 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.669482 (+ 1056us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.669487 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.670419 (+ 932us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.670429 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.671387 (+ 958us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.671404 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.671483 (+ 79us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.671538 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"spinlock_wait_cycles":9472,"thread_start_us":91,"threads_started":1}
I20260501 02:03:27.671828 27253 leader_election.cc:304] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102caed678ea47eba2d0eac75d34e8e5, 13c5181638b44f9691f1c216d58551a2; no voters:
I20260501 02:03:27.672250 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a94778c66a745a38b360eaba2dd52ec" candidate_uuid: "13c5181638b44f9691f1c216d58551a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9" is_pre_election: true
I20260501 02:03:27.672382 27544 raft_consensus.cc:2804] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:27.672392 27179 raft_consensus.cc:2468] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c5181638b44f9691f1c216d58551a2 in term 0.
I20260501 02:03:27.672435 27544 raft_consensus.cc:493] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:27.672472 27544 raft_consensus.cc:3060] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.673416 27544 raft_consensus.cc:515] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.673718 27544 leader_election.cc:290] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [CANDIDATE]: Term 1 election: Requested vote from peers c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283), 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119)
I20260501 02:03:27.674028 27179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a94778c66a745a38b360eaba2dd52ec" candidate_uuid: "13c5181638b44f9691f1c216d58551a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c578cc17dba44a8eb8115c3edd2878f9"
I20260501 02:03:27.674137 27179 raft_consensus.cc:3060] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.674203 27459 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a94778c66a745a38b360eaba2dd52ec" candidate_uuid: "13c5181638b44f9691f1c216d58551a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102caed678ea47eba2d0eac75d34e8e5"
I20260501 02:03:27.674299 27459 raft_consensus.cc:3060] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:27.675196 27179 raft_consensus.cc:2468] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c5181638b44f9691f1c216d58551a2 in term 1.
I20260501 02:03:27.675369 27459 raft_consensus.cc:2468] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c5181638b44f9691f1c216d58551a2 in term 1.
I20260501 02:03:27.675556 27252 leader_election.cc:304] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c5181638b44f9691f1c216d58551a2, c578cc17dba44a8eb8115c3edd2878f9; no voters:
I20260501 02:03:27.675766 27544 raft_consensus.cc:2804] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:27.675943 27544 raft_consensus.cc:697] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 1 LEADER]: Becoming Leader. State: Replica: 13c5181638b44f9691f1c216d58551a2, State: Running, Role: LEADER
I20260501 02:03:27.676180 27544 consensus_queue.cc:237] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } }
I20260501 02:03:27.678289 27019 catalog_manager.cc:5671] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c5181638b44f9691f1c216d58551a2 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "13c5181638b44f9691f1c216d58551a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13c5181638b44f9691f1c216d58551a2" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 34925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20260501 02:03:27.707990 27628 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.704707 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.18.96.194:34925)
0501 02:03:27.705028 (+ 321us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.705033 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.705238 (+ 205us) server_negotiation.cc:408] Connection header received
0501 02:03:27.705316 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.705319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.705367 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.705476 (+ 109us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.705807 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.706349 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.706962 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.707159 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.707207 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.707552 (+ 345us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.707680 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.707744 (+ 64us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.707804 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":152,"threads_started":1}
I20260501 02:03:27.717303 27584 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.714399 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34661 (local address 127.18.96.193:45283)
0501 02:03:27.714508 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.714511 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.714754 (+ 243us) server_negotiation.cc:408] Connection header received
0501 02:03:27.714870 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.714872 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.714923 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.714986 (+ 63us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.715432 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.715870 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.716779 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.716925 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.716952 (+ 27us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.717019 (+ 67us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.717071 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.717185 (+ 114us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.717216 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:03:27.717398 27631 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.714324 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.18.96.194:34661)
0501 02:03:27.714673 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.714687 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.714783 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.715014 (+ 231us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.715018 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.715028 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:03:27.715327 (+ 299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.715333 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.715984 (+ 651us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.715989 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.716675 (+ 686us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.716684 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.716794 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.716828 (+ 34us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.717098 (+ 270us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.717104 (+ 6us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.717166 (+ 62us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.717228 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
I20260501 02:03:27.722831 27179 raft_consensus.cc:1275] T 1a94778c66a745a38b360eaba2dd52ec P c578cc17dba44a8eb8115c3edd2878f9 [term 1 FOLLOWER]: Refusing update from remote peer 13c5181638b44f9691f1c216d58551a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.722858 27459 raft_consensus.cc:1275] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Refusing update from remote peer 13c5181638b44f9691f1c216d58551a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:27.723276 27544 consensus_queue.cc:1048] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102caed678ea47eba2d0eac75d34e8e5" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.723424 27626 consensus_queue.cc:1048] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c578cc17dba44a8eb8115c3edd2878f9" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:27.740324 27592 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.737488 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:34925 (local address 127.18.96.193:42627)
0501 02:03:27.737624 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.737636 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:27.737701 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:27.738000 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:27.738004 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:27.738015 (+ 11us) client_negotiation.cc:190] Negotiated authn=TOKEN
0501 02:03:27.738236 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.738242 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.739177 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.739179 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:27.739549 (+ 370us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:27.739553 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.739691 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.739720 (+ 29us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.740045 (+ 325us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.740051 (+ 6us) client_negotiation.cc:770] Sending connection context
0501 02:03:27.740121 (+ 70us) client_negotiation.cc:241] Negotiation successful
0501 02:03:27.740180 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260501 02:03:27.740315 27628 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.737571 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:42627 (local address 127.18.96.194:34925)
0501 02:03:27.737695 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:27.737698 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:27.737709 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:27.737799 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:27.737803 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:27.737863 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:27.737943 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:27.738624 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.739090 (+ 466us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:27.739657 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:27.739856 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:27.739901 (+ 45us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:27.739958 (+ 57us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:27.740011 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:27.740141 (+ 130us) server_negotiation.cc:300] Negotiation successful
0501 02:03:27.740181 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:03:27.758082 27019 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.758206 27019 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:27.759996 27019 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 87c24bc67e174de4851812542a372920: Sending DeleteTablet for 3 replicas of tablet 1a94778c66a745a38b360eaba2dd52ec
I20260501 02:03:27.760603 27301 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a94778c66a745a38b360eaba2dd52ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42174
I20260501 02:03:27.760797 27443 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a94778c66a745a38b360eaba2dd52ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:27 UTC) from {username='oryx', principal='oryx/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260501 02:03:27.760866 27597 tablet_replica.cc:333] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: stopping tablet replica
I20260501 02:03:27.760950 27599 tablet_replica.cc:333] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: stopping tablet replica
I20260501 02:03:27.760982 27597 raft_consensus.cc:2243] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:27.761052 27599 raft_consensus.cc:2243] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:27.761153 27599 raft_consensus.cc:2272] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.761240 27597 raft_consensus.cc:2272] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:27.761794 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27094
I20260501 02:03:27.761876 27599 ts_tablet_manager.cc:1916] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.761934 27597 ts_tablet_manager.cc:1916] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:27.764657 27599 ts_tablet_manager.cc:1929] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:03:27.764745 27599 log.cc:1199] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/1a94778c66a745a38b360eaba2dd52ec
I20260501 02:03:27.764955 27599 ts_tablet_manager.cc:1950] T 1a94778c66a745a38b360eaba2dd52ec P 102caed678ea47eba2d0eac75d34e8e5: Deleting consensus metadata
I20260501 02:03:27.764986 27597 ts_tablet_manager.cc:1929] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260501 02:03:27.765048 27597 log.cc:1199] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/1a94778c66a745a38b360eaba2dd52ec
I20260501 02:03:27.765291 27597 ts_tablet_manager.cc:1950] T 1a94778c66a745a38b360eaba2dd52ec P 13c5181638b44f9691f1c216d58551a2: Deleting consensus metadata
I20260501 02:03:27.765532 27005 catalog_manager.cc:5002] TS 102caed678ea47eba2d0eac75d34e8e5 (127.18.96.195:40119): tablet 1a94778c66a745a38b360eaba2dd52ec (table test-table [id=4fd62acfad6b4ff78ceaa5916a6a0963]) successfully deleted
I20260501 02:03:27.765923 27007 catalog_manager.cc:5002] TS 13c5181638b44f9691f1c216d58551a2 (127.18.96.194:34925): tablet 1a94778c66a745a38b360eaba2dd52ec (table test-table [id=4fd62acfad6b4ff78ceaa5916a6a0963]) successfully deleted
W20260501 02:03:27.771490 27004 connection.cc:570] client connection to 127.18.96.193:45283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260501 02:03:27.771597 27004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 02:03:27.772246 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27236
I20260501 02:03:27.772588 27637 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:27.772148 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:45283 (local address 127.0.0.1:45600)
0501 02:03:27.772433 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:27.772493 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:45283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":61,"threads_started":1}
W20260501 02:03:27.773809 27004 catalog_manager.cc:4729] TS c578cc17dba44a8eb8115c3edd2878f9 (127.18.96.193:45283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a94778c66a745a38b360eaba2dd52ec: Network error: Client connection negotiation failed: client connection to 127.18.96.193:45283: connect: Connection refused (error 111)
I20260501 02:03:27.778512 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27378
I20260501 02:03:27.784921 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 26988
2026-05-01T02:03:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4077 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[27640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[27640](info): set up 2 sockets
May 01 02:03:27 dist-test-slave-2j90 krb5kdc[27640](info): commencing operation
krb5kdc: starting...
W20260501 02:03:29.750963 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:29 dist-test-slave-2j90 krb5kdc[27640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:29.880225 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44037
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:29.977944 27656 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:29.978251 27656 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:29.978349 27656 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:29.981531 27656 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:29.981616 27656 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:29.981655 27656 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:29.981731 27656 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:29.981756 27656 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:29.985865 27656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:29.986901 27656 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:29.987712 27656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:29.992693 27661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:29.992789 27662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:29.992736 27664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:29.992951 27656 server_base.cc:1061] running on GCE node
I20260501 02:03:29.993387 27656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:29.994213 27656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:29.995389 27656 hybrid_clock.cc:648] HybridClock initialized: now 1777601009995370 us; error 40 us; skew 500 ppm
W20260501 02:03:29.998153 27656 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-05-01T02:03:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2220 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:30 dist-test-slave-2j90 krb5kdc[27669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:30 dist-test-slave-2j90 krb5kdc[27669](info): set up 2 sockets
May 01 02:03:30 dist-test-slave-2j90 krb5kdc[27669](info): commencing operation
krb5kdc: starting...
W20260501 02:03:31.959133 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:32.088922 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38985
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:32.185832 27685 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:32.186043 27685 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:32.186151 27685 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:32.189193 27685 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:32.189241 27685 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:32.189262 27685 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:32.189280 27685 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:32.189296 27685 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:32.193256 27685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:32.194255 27685 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:32.195039 27685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:32.200410 27693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.200415 27690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.200611 27691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:32.201058 27685 server_base.cc:1061] running on GCE node
I20260501 02:03:32.201536 27685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:32.202484 27685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:32.203625 27685 hybrid_clock.cc:648] HybridClock initialized: now 1777601012203610 us; error 30 us; skew 500 ppm
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:32.206156 27685 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:03:32.207142 27685 webserver.cc:492] Webserver started at http://127.18.96.254:39055/ using document root <none> and password file <none>
I20260501 02:03:32.207659 27685 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:32.207729 27685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:32.207939 27685 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:32.209553 27685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "0addc51dd7bf4bf5ab17110cf22eb5af"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "7194b3613191c33fc87be635e365f3b0"
server_key_iv: "50eb63c319b988e81eeaa217c776db8d"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.210093 27685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "0addc51dd7bf4bf5ab17110cf22eb5af"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "7194b3613191c33fc87be635e365f3b0"
server_key_iv: "50eb63c319b988e81eeaa217c776db8d"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.213109 27685 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:03:32.215171 27700 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.216081 27685 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:32.216217 27685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "0addc51dd7bf4bf5ab17110cf22eb5af"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "7194b3613191c33fc87be635e365f3b0"
server_key_iv: "50eb63c319b988e81eeaa217c776db8d"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.216313 27685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:32.246858 27685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:32.249485 27685 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:32.249686 27685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:32.257169 27685 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38985
I20260501 02:03:32.257174 27752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38985 every 8 connection(s)
I20260501 02:03:32.258263 27685 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:32.260866 27753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:32.264322 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27685
I20260501 02:03:32.264454 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:32.264729 18819 external_mini_cluster.cc:1468] Setting key 5bbe994b1bbbe915e251cc1fc94fd99a
I20260501 02:03:32.266443 27753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Bootstrap starting.
I20260501 02:03:32.268918 27753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:32.269670 27753 log.cc:826] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:32.271589 27753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: No bootstrap required, opened a new log
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:32.274484 27753 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:32.274695 27753 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:32.274755 27753 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0addc51dd7bf4bf5ab17110cf22eb5af, State: Initialized, Role: FOLLOWER
I20260501 02:03:32.275206 27753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:32.275364 27753 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:32.275436 27753 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:32.275537 27753 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:32.276528 27753 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:32.276957 27753 leader_election.cc:304] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0addc51dd7bf4bf5ab17110cf22eb5af; no voters:
I20260501 02:03:32.277304 27753 leader_election.cc:290] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:32.277616 27758 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:32.277807 27758 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 LEADER]: Becoming Leader. State: Replica: 0addc51dd7bf4bf5ab17110cf22eb5af, State: Running, Role: LEADER
I20260501 02:03:32.278031 27756 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.266020 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.18.96.254:38985)
0501 02:03:32.266415 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:32.266426 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:32.266475 (+ 49us) server_negotiation.cc:408] Connection header received
0501 02:03:32.267156 (+ 681us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:32.267179 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:32.267504 (+ 325us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:32.267844 (+ 340us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:32.268611 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.269749 (+ 1138us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.270602 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.270862 (+ 260us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.273184 (+ 2322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:32.273201 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:32.273212 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:32.273238 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:32.275358 (+ 2120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.275849 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.275854 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.275858 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.275926 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.276184 (+ 258us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.276186 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.276187 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.276470 (+ 283us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:32.276643 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:32.276964 (+ 321us) server_negotiation.cc:300] Negotiation successful
0501 02:03:32.277206 (+ 242us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":98,"threads_started":1}
I20260501 02:03:32.278123 27758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:32.278483 27753 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:32.279541 27759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } } }
I20260501 02:03:32.279642 27759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:32.279922 27760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0addc51dd7bf4bf5ab17110cf22eb5af. Latest consensus state: current_term: 1 leader_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } } }
I20260501 02:03:32.280001 27760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:32.281023 27767 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:32.283390 27767 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:32.288331 27767 catalog_manager.cc:1357] Generated new cluster ID: f9715c52ec744039842c49998a3c94e0
I20260501 02:03:32.288407 27767 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:32.319794 27767 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:32.320915 27767 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:32.333710 27767 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Generated new TSK 0
I20260501 02:03:32.334334 27767 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:32.389086 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38985
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:32.487534 27781 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:32.487716 27781 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:32.487782 27781 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:32.490847 27781 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:32.490909 27781 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:32.490980 27781 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:32.494647 27781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:32.495471 27781 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:32.496151 27781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:32.502270 27786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.502326 27787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.502501 27789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:32.502377 27781 server_base.cc:1061] running on GCE node
I20260501 02:03:32.502856 27781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:32.503329 27781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:32.504503 27781 hybrid_clock.cc:648] HybridClock initialized: now 1777601012504480 us; error 35 us; skew 500 ppm
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:32.506966 27781 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:03:32.507923 27781 webserver.cc:492] Webserver started at http://127.18.96.193:44979/ using document root <none> and password file <none>
I20260501 02:03:32.508451 27781 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:32.508496 27781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:32.508690 27781 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:32.510267 27781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "d52378bec1034cfb80765d97b586c3e5"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "56452eb0e08720064ceb0eb0cfa0c4f2"
server_key_iv: "3f92beb913c8a813f6892b966b8538f2"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.510701 27781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "d52378bec1034cfb80765d97b586c3e5"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "56452eb0e08720064ceb0eb0cfa0c4f2"
server_key_iv: "3f92beb913c8a813f6892b966b8538f2"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.513672 27781 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:32.515636 27796 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.516499 27781 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:32.516605 27781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "d52378bec1034cfb80765d97b586c3e5"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "56452eb0e08720064ceb0eb0cfa0c4f2"
server_key_iv: "3f92beb913c8a813f6892b966b8538f2"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.516691 27781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:32.538337 27781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:32.541153 27781 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:32.541347 27781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:32.541924 27781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:32.542774 27781 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:32.542820 27781 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.542860 27781 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:32.542876 27781 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.551826 27781 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:36321
I20260501 02:03:32.551939 27909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:36321 every 8 connection(s)
I20260501 02:03:32.552749 27781 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:32.554209 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27781
I20260501 02:03:32.554363 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:32.554623 18819 external_mini_cluster.cc:1468] Setting key 7c6f049acaad0a2c66c1249ae58aeed8
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:03:32.565551 27756 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.554495 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:42255 (local address 127.18.96.254:38985)
0501 02:03:32.554625 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:32.554629 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:32.555418 (+ 789us) server_negotiation.cc:408] Connection header received
0501 02:03:32.556259 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:32.556263 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:32.556312 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:32.556405 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:32.558439 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.558944 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.559599 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.559787 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.562157 (+ 2370us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:32.562173 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:32.562175 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:32.562197 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:32.563648 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.564126 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.564130 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.564132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.564182 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.564453 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.564457 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.564459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.564652 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:32.564750 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:32.565226 (+ 476us) server_negotiation.cc:300] Negotiation successful
0501 02:03:32.565316 (+ 90us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260501 02:03:32.566071 27912 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.554820 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:42255)
0501 02:03:32.555288 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:32.555325 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:32.556082 (+ 757us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:32.556529 (+ 447us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:32.556535 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:32.556905 (+ 370us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:32.558277 (+ 1372us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.558298 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.559082 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.559090 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:32.559478 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.559485 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.559683 (+ 198us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.560271 (+ 588us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:32.560290 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:32.562007 (+ 1717us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:32.563773 (+ 1766us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.563778 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.563788 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.564020 (+ 232us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.564276 (+ 256us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.564283 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.564285 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.564364 (+ 79us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.564802 (+ 438us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:32.564806 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:32.564990 (+ 184us) client_negotiation.cc:770] Sending connection context
0501 02:03:32.565167 (+ 177us) client_negotiation.cc:241] Negotiation successful
0501 02:03:32.565366 (+ 199us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:03:32.567183 27910 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38985
I20260501 02:03:32.567402 27910 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:32.567829 27910 heartbeater.cc:507] Master 127.18.96.254:38985 requested a full tablet report, sending...
I20260501 02:03:32.569134 27717 ts_manager.cc:194] Registered new tserver with Master: d52378bec1034cfb80765d97b586c3e5 (127.18.96.193:36321)
I20260501 02:03:32.570268 27717 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:42255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:32.601245 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38985
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:32.696674 27917 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:32.696844 27917 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:32.696893 27917 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:32.700006 27917 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:32.700055 27917 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:32.700132 27917 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:32.703972 27917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:32.704851 27917 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:32.705591 27917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 02:03:32.711720 27917 server_base.cc:1061] running on GCE node
W20260501 02:03:32.711613 27922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.711616 27925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.711732 27923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:32.712218 27917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:32.712721 27917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:32.713889 27917 hybrid_clock.cc:648] HybridClock initialized: now 1777601012713885 us; error 29 us; skew 500 ppm
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:32.716487 27917 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:03:32.717422 27917 webserver.cc:492] Webserver started at http://127.18.96.194:42285/ using document root <none> and password file <none>
I20260501 02:03:32.717916 27917 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:32.718008 27917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:32.718237 27917 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:32.719785 27917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "0a78cd25bcd145c98a2e118c40570c07"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "2755be7c2beb54fdec0fd4c64cd77d43"
server_key_iv: "5c983f85e47fdb1481a322860ed7b72a"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.720379 27917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "0a78cd25bcd145c98a2e118c40570c07"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "2755be7c2beb54fdec0fd4c64cd77d43"
server_key_iv: "5c983f85e47fdb1481a322860ed7b72a"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.723536 27917 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260501 02:03:32.725488 27932 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.726456 27917 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:32.726538 27917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "0a78cd25bcd145c98a2e118c40570c07"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "2755be7c2beb54fdec0fd4c64cd77d43"
server_key_iv: "5c983f85e47fdb1481a322860ed7b72a"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.726603 27917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:32.738720 27917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:32.741231 27917 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:32.741362 27917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:32.741777 27917 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:32.742569 27917 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:32.742614 27917 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.742650 27917 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:32.742671 27917 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.751361 27917 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:38641
I20260501 02:03:32.751386 28045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:38641 every 8 connection(s)
I20260501 02:03:32.752236 27917 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:32.755717 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 27917
I20260501 02:03:32.755864 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:32.756157 18819 external_mini_cluster.cc:1468] Setting key 0d7f945601c17ed7c625feec66fd5769
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:32.764757 27756 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.753840 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:45415 (local address 127.18.96.254:38985)
0501 02:03:32.754017 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:32.754021 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:32.754631 (+ 610us) server_negotiation.cc:408] Connection header received
0501 02:03:32.755390 (+ 759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:32.755393 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:32.755438 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:32.755512 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:32.757277 (+ 1765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.758014 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.758670 (+ 656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.758899 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.761101 (+ 2202us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:32.761128 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:32.761136 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:32.761171 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:32.762873 (+ 1702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.763353 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.763357 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.763360 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.763415 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.763693 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.763697 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.763699 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.763891 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:32.764002 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:32.764436 (+ 434us) server_negotiation.cc:300] Negotiation successful
0501 02:03:32.764577 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:03:32.765228 28048 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.754132 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:45415)
0501 02:03:32.754498 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:32.754527 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:32.755218 (+ 691us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:32.755702 (+ 484us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:32.755714 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:32.756227 (+ 513us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:32.757120 (+ 893us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.757132 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.758147 (+ 1015us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.758150 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:32.758559 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.758565 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.758708 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.759246 (+ 538us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:32.759274 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:32.760913 (+ 1639us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:32.763011 (+ 2098us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.763016 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.763025 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.763250 (+ 225us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.763516 (+ 266us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.763518 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.763520 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.763601 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.763996 (+ 395us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:32.764000 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:32.764191 (+ 191us) client_negotiation.cc:770] Sending connection context
0501 02:03:32.764360 (+ 169us) client_negotiation.cc:241] Negotiation successful
0501 02:03:32.764548 (+ 188us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:03:32.766434 28046 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38985
I20260501 02:03:32.766741 28046 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:32.767289 28046 heartbeater.cc:507] Master 127.18.96.254:38985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:03:32.768405 27717 ts_manager.cc:194] Registered new tserver with Master: 0a78cd25bcd145c98a2e118c40570c07 (127.18.96.194:38641)
I20260501 02:03:32.768967 27717 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:45415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:32.802351 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38985
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:32.901316 28053 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:32.901511 28053 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:32.901605 28053 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:32.904894 28053 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:32.904948 28053 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:32.905082 28053 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:32.908962 28053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:32.909915 28053 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:32.910734 28053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:32.916401 28058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.916409 28059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:32.916401 28061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:32.916612 28053 server_base.cc:1061] running on GCE node
I20260501 02:03:32.917021 28053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:32.917564 28053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:32.918743 28053 hybrid_clock.cc:648] HybridClock initialized: now 1777601012918731 us; error 33 us; skew 500 ppm
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:32.921355 28053 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:03:32.922281 28053 webserver.cc:492] Webserver started at http://127.18.96.195:38897/ using document root <none> and password file <none>
I20260501 02:03:32.922747 28053 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:32.922791 28053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:32.922968 28053 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:32.924571 28053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "1aa97861ee91443987f8cbb7607470fa"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "76ab1bff1e52fbb9d0da2f28cabd38f7"
server_key_iv: "a873d2c908e1a68059186046a43fea26"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.925026 28053 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "1aa97861ee91443987f8cbb7607470fa"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "76ab1bff1e52fbb9d0da2f28cabd38f7"
server_key_iv: "a873d2c908e1a68059186046a43fea26"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.928298 28053 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:32.930281 28068 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.931308 28053 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:32.931412 28053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "1aa97861ee91443987f8cbb7607470fa"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "76ab1bff1e52fbb9d0da2f28cabd38f7"
server_key_iv: "a873d2c908e1a68059186046a43fea26"
server_key_version: "encryptionkey@0"
I20260501 02:03:32.931495 28053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:32.956204 28053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:32.958611 28053 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:32.958799 28053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:32.959328 28053 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:32.960181 28053 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:32.960227 28053 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.960291 28053 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:32.960325 28053 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:32.968914 28053 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:39461
I20260501 02:03:32.968942 28181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:39461 every 8 connection(s)
I20260501 02:03:32.969755 28053 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:03:32 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601012, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:32.977363 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28053
I20260501 02:03:32.977491 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:32.977752 18819 external_mini_cluster.cc:1468] Setting key 5c8131d53478d193faf00502e09712dd
I20260501 02:03:32.981400 27756 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.971314 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:49487 (local address 127.18.96.254:38985)
0501 02:03:32.971447 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:32.971450 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:32.972006 (+ 556us) server_negotiation.cc:408] Connection header received
0501 02:03:32.972817 (+ 811us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:32.972821 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:32.972897 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:32.972999 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:32.974548 (+ 1549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.975042 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.975691 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.975869 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.978006 (+ 2137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:32.978025 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:32.978028 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:32.978054 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:32.979626 (+ 1572us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.980088 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.980092 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.980094 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.980138 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:32.980457 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:32.980460 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:32.980462 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:32.980598 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:32.980709 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:32.981134 (+ 425us) server_negotiation.cc:300] Negotiation successful
0501 02:03:32.981253 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:03:32.981880 28184 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:32.971494 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:49487)
0501 02:03:32.971877 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:32.971907 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:32.972626 (+ 719us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:32.973145 (+ 519us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:32.973153 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:32.973476 (+ 323us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:32.974403 (+ 927us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.974416 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.975169 (+ 753us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:32.975175 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:32.975571 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:32.975577 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:32.975775 (+ 198us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:32.976364 (+ 589us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:32.976381 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:32.977837 (+ 1456us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:32.979720 (+ 1883us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.979725 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.979735 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.979998 (+ 263us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.980263 (+ 265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:32.980266 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:32.980272 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:32.980360 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:32.980722 (+ 362us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:32.980729 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:32.980911 (+ 182us) client_negotiation.cc:770] Sending connection context
0501 02:03:32.981085 (+ 174us) client_negotiation.cc:241] Negotiation successful
0501 02:03:32.981300 (+ 215us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260501 02:03:32.982779 28182 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38985
I20260501 02:03:32.983034 28182 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:32.983492 28182 heartbeater.cc:507] Master 127.18.96.254:38985 requested a full tablet report, sending...
I20260501 02:03:32.984383 27717 ts_manager.cc:194] Registered new tserver with Master: 1aa97861ee91443987f8cbb7607470fa (127.18.96.195:39461)
I20260501 02:03:32.984910 27717 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:49487
I20260501 02:03:32.991274 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.193.keytab.
Entry for principal oryx/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.193.keytab.
WARNING: no policy specified for oryx/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.194.keytab.
Entry for principal oryx/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.194.keytab.
WARNING: no policy specified for oryx/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.195.keytab.
Entry for principal oryx/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.195.keytab.
I20260501 02:03:33.052250 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27781
I20260501 02:03:33.057928 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27917
I20260501 02:03:33.063416 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28053
I20260501 02:03:33.068763 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 27685
I20260501 02:03:33.074868 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38985
--webserver_interface=127.18.96.254
--webserver_port=39055
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:33.172777 28191 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:33.172977 28191 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:33.173048 28191 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:33.176265 28191 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:33.176316 28191 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:33.176337 28191 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:33.176355 28191 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:33.176419 28191 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:33.180214 28191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=39055
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:33.181099 28191 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:33.181861 28191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:33.186954 28197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.186961 28199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.186964 28196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:33.187415 28191 server_base.cc:1061] running on GCE node
I20260501 02:03:33.187789 28191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:33.188553 28191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:33.189695 28191 hybrid_clock.cc:648] HybridClock initialized: now 1777601013189674 us; error 36 us; skew 500 ppm
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601013, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:33.192137 28191 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:03:33.193038 28191 webserver.cc:492] Webserver started at http://127.18.96.254:39055/ using document root <none> and password file <none>
I20260501 02:03:33.193498 28191 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:33.193550 28191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:33.196555 28191 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:33.198239 28206 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.199245 28191 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:33.199371 28191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "0addc51dd7bf4bf5ab17110cf22eb5af"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "7194b3613191c33fc87be635e365f3b0"
server_key_iv: "50eb63c319b988e81eeaa217c776db8d"
server_key_version: "encryptionkey@0"
I20260501 02:03:33.199738 28191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:33.208714 28191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:33.210935 28191 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:33.211124 28191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:33.218067 28191 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38985
I20260501 02:03:33.218071 28258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38985 every 8 connection(s)
I20260501 02:03:33.218986 28191 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:33.219717 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28191
I20260501 02:03:33.220180 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:36321
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:38985
--webserver_port=44979
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:03:33.223496 28259 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:03:33.224949 28259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Bootstrap starting.
I20260501 02:03:33.240343 28259 log.cc:826] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:33.246441 28259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:03:33.246817 28259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Bootstrap complete.
I20260501 02:03:33.250038 28259 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:33.250327 28259 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0addc51dd7bf4bf5ab17110cf22eb5af, State: Initialized, Role: FOLLOWER
I20260501 02:03:33.250752 28259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:33.250857 28259 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:33.250931 28259 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:33.251014 28259 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:03:33.252444 28259 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:33.252777 28259 leader_election.cc:304] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0addc51dd7bf4bf5ab17110cf22eb5af; no voters:
I20260501 02:03:33.253113 28259 leader_election.cc:290] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 02:03:33.253192 28263 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 2 FOLLOWER]: Leader election won for term 2
I20260501 02:03:33.253364 28263 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [term 2 LEADER]: Becoming Leader. State: Replica: 0addc51dd7bf4bf5ab17110cf22eb5af, State: Running, Role: LEADER
I20260501 02:03:33.253695 28263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } }
I20260501 02:03:33.254168 28259 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:33.254528 28263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0addc51dd7bf4bf5ab17110cf22eb5af. Latest consensus state: current_term: 2 leader_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } } }
I20260501 02:03:33.254626 28263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:33.255204 28264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0addc51dd7bf4bf5ab17110cf22eb5af" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38985 } } }
I20260501 02:03:33.255321 28264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:33.255779 28267 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:33.258525 28267 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:33.259125 28267 catalog_manager.cc:1269] Loaded cluster ID: f9715c52ec744039842c49998a3c94e0
I20260501 02:03:33.259181 28267 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:33.261314 28267 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:33.261937 28267 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 0addc51dd7bf4bf5ab17110cf22eb5af: Loaded TSK: 0
I20260501 02:03:33.262809 28267 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:03:33.317986 28261 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:33.318212 28261 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:33.318357 28261 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:33.321442 28261 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:33.321491 28261 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:33.321631 28261 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:33.325618 28261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.193.keytab
--principal=oryx/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:36321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=44979
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:33.326581 28261 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:33.327436 28261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:33.332868 28285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.332921 28286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.332995 28288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:33.333611 28261 server_base.cc:1061] running on GCE node
I20260501 02:03:33.333989 28261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:33.334534 28261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:33.335723 28261 hybrid_clock.cc:648] HybridClock initialized: now 1777601013335677 us; error 39 us; skew 500 ppm
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601013, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:33.338622 28261 init.cc:377] Logged in from keytab as oryx/127.18.96.193@KRBTEST.COM (short username oryx)
I20260501 02:03:33.339725 28261 webserver.cc:492] Webserver started at http://127.18.96.193:44979/ using document root <none> and password file <none>
I20260501 02:03:33.340334 28261 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:33.340404 28261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:33.343992 28261 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260501 02:03:33.345876 28295 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.347018 28261 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:33.347127 28261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "d52378bec1034cfb80765d97b586c3e5"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "56452eb0e08720064ceb0eb0cfa0c4f2"
server_key_iv: "3f92beb913c8a813f6892b966b8538f2"
server_key_version: "encryptionkey@0"
I20260501 02:03:33.347568 28261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:33.373477 28261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:33.376518 28261 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:33.376706 28261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:33.377360 28261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:33.378417 28261 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:33.378475 28261 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.378526 28261 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:33.378556 28261 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.389833 28261 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:36321
I20260501 02:03:33.389868 28408 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:36321 every 8 connection(s)
I20260501 02:03:33.390990 28261 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:33.395615 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28261
I20260501 02:03:33.396062 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:38641
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:38985
--webserver_port=42285
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.402932 28412 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.392991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:33749)
0501 02:03:33.393491 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.393530 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.394450 (+ 920us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.395372 (+ 922us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.395381 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.395864 (+ 483us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.396941 (+ 1077us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.396957 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.398195 (+ 1238us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.398199 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.398738 (+ 539us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.398747 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.398941 (+ 194us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.399587 (+ 646us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.399608 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.402227 (+ 2619us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":120,"threads_started":1}
W20260501 02:03:33.403190 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.392981 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:33749 (local address 127.18.96.254:38985)
0501 02:03:33.394173 (+ 1192us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.394181 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.394213 (+ 32us) server_negotiation.cc:408] Connection header received
0501 02:03:33.394777 (+ 564us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.394786 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.395007 (+ 221us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.395221 (+ 214us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.397141 (+ 1920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.398081 (+ 940us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.398876 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.399161 (+ 285us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.402428 (+ 3267us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:33749: BlockingRecv error: recv got EOF from 127.18.96.193:33749 (error 108)
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":114,"threads_started":1}
W20260501 02:03:33.404100 28409 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:38985 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.254:38985: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.409780 28412 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.404561 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:38907)
0501 02:03:33.404817 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.404831 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.404904 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.405216 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.405220 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.405399 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.405634 (+ 235us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.405642 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.406749 (+ 1107us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.406753 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.407274 (+ 521us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.407282 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.407392 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.407867 (+ 475us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.407882 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.409679 (+ 1797us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180}
W20260501 02:03:33.409909 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.404834 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:38907 (local address 127.18.96.254:38985)
0501 02:03:33.404968 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.404975 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.404990 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:33.405029 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.405035 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.405083 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.405175 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.405769 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.406612 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.407394 (+ 782us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.407935 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.409804 (+ 1869us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:38907: BlockingRecv error: recv got EOF from 127.18.96.193:38907 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.415345 28412 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.410478 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:57435)
0501 02:03:33.410714 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.410726 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.410791 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.411057 (+ 266us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.411061 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.411204 (+ 143us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.411380 (+ 176us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.411387 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.412302 (+ 915us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.412306 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.412782 (+ 476us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.412790 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.412900 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.413334 (+ 434us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.413350 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.415239 (+ 1889us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260501 02:03:33.415453 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.410713 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57435 (local address 127.18.96.254:38985)
0501 02:03:33.410814 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.410817 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.410856 (+ 39us) server_negotiation.cc:408] Connection header received
0501 02:03:33.410900 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.410903 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.410937 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.411012 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.411498 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.412181 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.412903 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.413352 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.415362 (+ 2010us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:57435: BlockingRecv error: recv got EOF from 127.18.96.193:57435 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:03:33.415721 28409 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:03:33.496532 28413 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:33.496694 28413 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:33.496786 28413 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:33.499841 28413 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:33.499889 28413 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:33.500017 28413 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:33.503983 28413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.194.keytab
--principal=oryx/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:38641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=42285
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:33.504825 28413 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:33.505554 28413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:33.511322 28418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.511329 28421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.511334 28419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:33.511649 28413 server_base.cc:1061] running on GCE node
I20260501 02:03:33.511945 28413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:33.512354 28413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:33.513499 28413 hybrid_clock.cc:648] HybridClock initialized: now 1777601013513484 us; error 28 us; skew 500 ppm
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601013, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:33.515823 28413 init.cc:377] Logged in from keytab as oryx/127.18.96.194@KRBTEST.COM (short username oryx)
I20260501 02:03:33.516717 28413 webserver.cc:492] Webserver started at http://127.18.96.194:42285/ using document root <none> and password file <none>
I20260501 02:03:33.517153 28413 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:33.517208 28413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:33.520179 28413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:33.521736 28428 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.522621 28413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:33.522756 28413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "0a78cd25bcd145c98a2e118c40570c07"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "2755be7c2beb54fdec0fd4c64cd77d43"
server_key_iv: "5c983f85e47fdb1481a322860ed7b72a"
server_key_version: "encryptionkey@0"
I20260501 02:03:33.523141 28413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:33.548899 28413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:33.551443 28413 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:33.551637 28413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:33.552187 28413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:33.553157 28413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:33.553222 28413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.553292 28413 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:33.553333 28413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.562569 28413 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:38641
I20260501 02:03:33.562611 28541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:38641 every 8 connection(s)
I20260501 02:03:33.563482 28413 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.571532 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28413
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.572002 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:39461
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:38985
--webserver_port=38897
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:33.572719 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.565071 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:42129 (local address 127.18.96.254:38985)
0501 02:03:33.565196 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.565200 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.565775 (+ 575us) server_negotiation.cc:408] Connection header received
0501 02:03:33.566637 (+ 862us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.566641 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.566707 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.566799 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.568209 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.568653 (+ 444us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.569300 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.569479 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.572559 (+ 3080us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:42129: BlockingRecv error: recv got EOF from 127.18.96.194:42129 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:03:33.572978 28544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.565274 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:42129)
0501 02:03:33.565652 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.565681 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.566454 (+ 773us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.566969 (+ 515us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.566975 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.567287 (+ 312us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.568072 (+ 785us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.568083 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.568785 (+ 702us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.568789 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.569185 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.569191 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.569380 (+ 189us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.570077 (+ 697us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.570095 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.572400 (+ 2305us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
W20260501 02:03:33.573431 28542 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:38985 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.254:38985: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.578979 28544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.573874 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:41211)
0501 02:03:33.574083 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.574099 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.574185 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.574421 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.574427 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.574598 (+ 171us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.574838 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.574848 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.575590 (+ 742us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.575597 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.576249 (+ 652us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.576261 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.576391 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.576839 (+ 448us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.576871 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.578811 (+ 1940us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260501 02:03:33.578989 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.573937 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:41211 (local address 127.18.96.254:38985)
0501 02:03:33.574062 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.574065 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.574143 (+ 78us) server_negotiation.cc:408] Connection header received
0501 02:03:33.574271 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.574273 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.574309 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.574375 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.574961 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.575460 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.576369 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.576493 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.578907 (+ 2414us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:41211: BlockingRecv error: recv got EOF from 127.18.96.194:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.583583 28544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.579561 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:53359)
0501 02:03:33.579685 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.579698 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.579776 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.580027 (+ 251us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.580030 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.580179 (+ 149us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.580305 (+ 126us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.580309 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.580936 (+ 627us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.580942 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.581269 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.581274 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.581407 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.581860 (+ 453us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.581872 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.583463 (+ 1591us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260501 02:03:33.583653 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.579619 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:53359 (local address 127.18.96.254:38985)
0501 02:03:33.579724 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.579729 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.579743 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:03:33.579884 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.579887 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.579935 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.580013 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.580411 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.580840 (+ 429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.581395 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.581590 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.583552 (+ 1962us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:53359: BlockingRecv error: recv got EOF from 127.18.96.194:53359 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:03:33.583997 28542 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:03:33.668946 28545 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:33.669155 28545 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:33.669248 28545 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:33.672395 28545 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:33.672477 28545 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:33.672578 28545 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:33.676592 28545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/krb5kdc/oryx_127.18.96.195.keytab
--principal=oryx/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:39461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=38897
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:33.677529 28545 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:33.678287 28545 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:33.684312 28550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:33.684312 28553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:33.684427 28545 server_base.cc:1061] running on GCE node
W20260501 02:03:33.684340 28551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:33.684860 28545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:33.685374 28545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:33.686512 28545 hybrid_clock.cc:648] HybridClock initialized: now 1777601013686492 us; error 33 us; skew 500 ppm
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601013, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:33.688931 28545 init.cc:377] Logged in from keytab as oryx/127.18.96.195@KRBTEST.COM (short username oryx)
I20260501 02:03:33.689826 28545 webserver.cc:492] Webserver started at http://127.18.96.195:38897/ using document root <none> and password file <none>
I20260501 02:03:33.690359 28545 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:33.690418 28545 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:33.693686 28545 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:03:33.695403 28560 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.696309 28545 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:33.696396 28545 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "1aa97861ee91443987f8cbb7607470fa"
format_stamp: "Formatted at 2026-05-01 02:03:32 on dist-test-slave-2j90"
server_key: "76ab1bff1e52fbb9d0da2f28cabd38f7"
server_key_iv: "a873d2c908e1a68059186046a43fea26"
server_key_version: "encryptionkey@0"
I20260501 02:03:33.696774 28545 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:33.720419 28545 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:33.722736 28545 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:33.722886 28545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:33.723363 28545 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:33.724185 28545 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:33.724231 28545 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.724268 28545 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:33.724290 28545 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:33.733134 28545 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:39461
I20260501 02:03:33.733163 28673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:39461 every 8 connection(s)
I20260501 02:03:33.734155 28545 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:33.737733 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28545
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260501 02:03:33.744048 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.735896 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:58941 (local address 127.18.96.254:38985)
0501 02:03:33.736082 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.736087 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.736716 (+ 629us) server_negotiation.cc:408] Connection header received
0501 02:03:33.737508 (+ 792us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.737512 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.737573 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.737672 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.739518 (+ 1846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.740229 (+ 711us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.740957 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.741118 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.743929 (+ 2811us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:58941: BlockingRecv error: recv got EOF from 127.18.96.195:58941 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260501 02:03:33.744379 28676 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.736198 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:58941)
0501 02:03:33.736582 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.736613 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.737298 (+ 685us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.737821 (+ 523us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.737834 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.738331 (+ 497us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.739377 (+ 1046us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.739389 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.740337 (+ 948us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.740341 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.740854 (+ 513us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.740868 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.740995 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.741532 (+ 537us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.741562 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.743784 (+ 2222us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
W20260501 02:03:33.744817 28674 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:38985 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.96.254:38985: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
I20260501 02:03:33.746909 28678 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.738625 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.18.96.254:38985)
0501 02:03:33.738901 (+ 276us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.738906 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.738923 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:03:33.738964 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.738967 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.739017 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.739128 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.740008 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.740636 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.741453 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.741641 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.742630 (+ 989us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:33.742648 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:33.742658 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:33.742684 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:33.744718 (+ 2034us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:33.745470 (+ 752us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:33.745474 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:33.745478 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:33.745544 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:33.745789 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:33.745792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:33.745794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:33.746119 (+ 325us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:33.746292 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:33.746592 (+ 300us) server_negotiation.cc:300] Negotiation successful
0501 02:03:33.746795 (+ 203us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.749629 28676 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.745221 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:33173)
0501 02:03:33.745476 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.745488 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.745568 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.745788 (+ 220us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.745791 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.745932 (+ 141us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.746148 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.746153 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.746832 (+ 679us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.746840 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.747278 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.747285 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.747391 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.747792 (+ 401us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.747803 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.749511 (+ 1708us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189}
W20260501 02:03:33.749701 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.745352 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:33173 (local address 127.18.96.254:38985)
0501 02:03:33.745453 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.745457 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.745545 (+ 88us) server_negotiation.cc:408] Connection header received
0501 02:03:33.745650 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.745653 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.745693 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.745774 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.746254 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.746733 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.747381 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.747566 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.749599 (+ 2033us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:33173: BlockingRecv error: recv got EOF from 127.18.96.195:33173 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:33 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:33.753862 28676 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:33.750143 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:40713)
0501 02:03:33.750262 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:33.750273 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:33.750331 (+ 58us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:33.750526 (+ 195us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:33.750528 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:33.750641 (+ 113us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:33.750773 (+ 132us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.750777 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.751396 (+ 619us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.751399 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:33.751728 (+ 329us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:33.751734 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.751818 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.752240 (+ 422us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:33.752252 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:33.753788 (+ 1536us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260501 02:03:33.754076 28411 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:33.750224 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:40713 (local address 127.18.96.254:38985)
0501 02:03:33.750327 (+ 103us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:33.750332 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:33.750349 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:03:33.750391 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:33.750393 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:33.750422 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:33.750501 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:33.750868 (+ 367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.751303 (+ 435us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:33.751815 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:33.751936 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:33.753905 (+ 1969us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:40713: BlockingRecv error: recv got EOF from 127.18.96.195:40713 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260501 02:03:33.754073 28674 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:34.420816 28679 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:34.416292 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:50825)
0501 02:03:34.416541 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:34.416553 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:34.416662 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:34.416886 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:34.416888 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:34.417067 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:34.417289 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.417295 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.417995 (+ 700us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.417997 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:34.418377 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.418382 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.418466 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.418992 (+ 526us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:34.419004 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:34.420707 (+ 1703us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":84,"threads_started":1}
W20260501 02:03:34.420969 28680 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:34.416341 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:50825 (local address 127.18.96.254:38985)
0501 02:03:34.416566 (+ 225us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:34.416569 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:34.416631 (+ 62us) server_negotiation.cc:408] Connection header received
0501 02:03:34.416741 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:34.416743 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:34.416788 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:34.416861 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:34.417394 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.417890 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.418466 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.418597 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.420816 (+ 2219us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:50825: BlockingRecv error: recv got EOF from 127.18.96.193:50825 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:34.589566 28681 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:34.584548 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:46809)
0501 02:03:34.584853 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:34.584892 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:34.585026 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:34.585312 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:34.585315 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:34.585529 (+ 214us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:34.585740 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.585746 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.586554 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.586561 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:34.586963 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.586969 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.587097 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.587668 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:34.587680 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:34.589453 (+ 1773us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
W20260501 02:03:34.589691 28680 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:34.584661 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46809 (local address 127.18.96.254:38985)
0501 02:03:34.584820 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:34.584824 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:34.584967 (+ 143us) server_negotiation.cc:408] Connection header received
0501 02:03:34.585134 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:34.585150 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:34.585203 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:34.585290 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:34.585863 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.586442 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.587081 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.587296 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.589582 (+ 2286us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:46809: BlockingRecv error: recv got EOF from 127.18.96.194:46809 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:34 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:34.759195 28682 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:34.754608 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:55083)
0501 02:03:34.754944 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:34.754957 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:34.755050 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:34.755274 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:34.755276 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:34.755443 (+ 167us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:34.755657 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.755662 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.756354 (+ 692us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.756357 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:34.756723 (+ 366us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:34.756729 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.756818 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.757389 (+ 571us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:34.757403 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:34.759087 (+ 1684us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":155,"threads_started":1}
W20260501 02:03:34.759313 28680 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:34.754685 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:55083 (local address 127.18.96.254:38985)
0501 02:03:34.754914 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:34.754917 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:34.755021 (+ 104us) server_negotiation.cc:408] Connection header received
0501 02:03:34.755123 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:34.755126 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:34.755173 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:34.755242 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:34.755752 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.756246 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:34.756820 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:34.756965 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:34.759204 (+ 2239us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:55083: BlockingRecv error: recv got EOF from 127.18.96.195:55083 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:35.430145 28683 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:35.421713 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:55673)
0501 02:03:35.422042 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:35.422055 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:35.422148 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:35.422407 (+ 259us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:35.422409 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:35.422575 (+ 166us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:35.422760 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.422764 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.423825 (+ 1061us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.423829 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:35.424371 (+ 542us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.424383 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.424490 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.428069 (+ 3579us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:35.428083 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:35.429979 (+ 1896us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":75,"threads_started":1}
W20260501 02:03:35.430199 28684 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:35.421834 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:55673 (local address 127.18.96.254:38985)
0501 02:03:35.422112 (+ 278us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:35.422115 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:35.422130 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:03:35.422242 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:35.422244 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:35.422290 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:35.422378 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:35.422903 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.423682 (+ 779us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.424507 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.424670 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.430078 (+ 5408us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:55673: BlockingRecv error: recv got EOF from 127.18.96.193:55673 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":74,"threads_started":1}
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:35.595404 28685 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:35.590457 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:53813)
0501 02:03:35.590733 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:35.590745 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:35.590878 (+ 133us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:35.591160 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:35.591162 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:35.591361 (+ 199us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:35.591543 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.591548 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.592296 (+ 748us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.592301 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:35.592680 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.592686 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.592807 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.593328 (+ 521us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:35.593340 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:35.595248 (+ 1908us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
W20260501 02:03:35.595436 28684 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:35.590501 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:53813 (local address 127.18.96.254:38985)
0501 02:03:35.590666 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:35.590670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:35.590819 (+ 149us) server_negotiation.cc:408] Connection header received
0501 02:03:35.591004 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:35.591015 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:35.591059 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:35.591139 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:35.591660 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.592185 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.592798 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.593007 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.595342 (+ 2335us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:53813: BlockingRecv error: recv got EOF from 127.18.96.194:53813 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:35 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:35.764515 28686 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:35.760003 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:56635)
0501 02:03:35.760284 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:35.760297 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:35.760390 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:35.760608 (+ 218us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:35.760610 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:35.760772 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:35.760978 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.760983 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.761664 (+ 681us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.761666 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:35.762103 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:35.762110 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.762199 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.762690 (+ 491us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:35.762702 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:35.764408 (+ 1706us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
W20260501 02:03:35.764638 28684 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:35.760140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:56635 (local address 127.18.96.254:38985)
0501 02:03:35.760274 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:35.760277 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:35.760358 (+ 81us) server_negotiation.cc:408] Connection header received
0501 02:03:35.760461 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:35.760463 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:35.760507 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:35.760573 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:35.761067 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.761574 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:35.762203 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:35.762329 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:35.764527 (+ 2198us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:56635: BlockingRecv error: recv got EOF from 127.18.96.195:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:36.435618 28687 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:36.431086 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:37619)
0501 02:03:36.431358 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:36.431370 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:36.431456 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:36.431671 (+ 215us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:36.431673 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:36.431835 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:36.432009 (+ 174us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.432013 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.432631 (+ 618us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.432633 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:36.433010 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.433015 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.433097 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.433597 (+ 500us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:36.433609 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:36.435443 (+ 1834us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
W20260501 02:03:36.435674 28688 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:36.431153 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:37619 (local address 127.18.96.254:38985)
0501 02:03:36.431401 (+ 248us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:36.431404 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:36.431416 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:03:36.431531 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:36.431533 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:36.431573 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:36.431643 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:36.432097 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.432551 (+ 454us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.433100 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.433226 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.435543 (+ 2317us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:37619: BlockingRecv error: recv got EOF from 127.18.96.193:37619 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:36.601171 28689 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:36.596266 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:32873)
0501 02:03:36.596540 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:36.596552 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:36.596683 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:36.596945 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:36.596948 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:36.597156 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:36.597370 (+ 214us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.597376 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.598162 (+ 786us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.598167 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:36.598549 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.598555 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.598682 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.599192 (+ 510us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:36.599205 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:36.601004 (+ 1799us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
W20260501 02:03:36.601207 28688 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:36.596293 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:32873 (local address 127.18.96.254:38985)
0501 02:03:36.596459 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:36.596464 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:36.596622 (+ 158us) server_negotiation.cc:408] Connection header received
0501 02:03:36.596778 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:36.596781 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:36.596835 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:36.596941 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:36.597488 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.598055 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.598665 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.598873 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.601109 (+ 2236us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:32873: BlockingRecv error: recv got EOF from 127.18.96.194:32873 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:36 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:36.770125 28690 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:36.765385 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:44861)
0501 02:03:36.765644 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:36.765658 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:36.765802 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:36.766143 (+ 341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:36.766149 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:36.766335 (+ 186us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:36.766526 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.766531 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.767219 (+ 688us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.767221 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:36.767586 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:36.767591 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.767673 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.768212 (+ 539us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:36.768224 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:36.769999 (+ 1775us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":87,"threads_started":1}
W20260501 02:03:36.770222 28688 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:36.765441 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:44861 (local address 127.18.96.254:38985)
0501 02:03:36.765595 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:36.765602 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:36.765867 (+ 265us) server_negotiation.cc:408] Connection header received
0501 02:03:36.765903 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:36.765906 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:36.766017 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:36.766100 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:36.766632 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.767123 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:36.767680 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:36.767819 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:36.770094 (+ 2275us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:44861: BlockingRecv error: recv got EOF from 127.18.96.195:44861 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:37.440891 28691 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:37.436400 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:39715)
0501 02:03:37.436668 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:37.436681 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:37.436780 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:37.437027 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:37.437029 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:37.437189 (+ 160us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:37.437387 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.437393 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.438113 (+ 720us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.438116 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:37.438476 (+ 360us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.438481 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.438569 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.439104 (+ 535us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:37.439115 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:37.440781 (+ 1666us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
W20260501 02:03:37.441043 28692 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:37.436456 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39715 (local address 127.18.96.254:38985)
0501 02:03:37.436695 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:37.436698 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:37.436751 (+ 53us) server_negotiation.cc:408] Connection header received
0501 02:03:37.436864 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:37.436866 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:37.436910 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:37.436985 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:37.437496 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.438013 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.438571 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.438711 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.440901 (+ 2190us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:39715: BlockingRecv error: recv got EOF from 127.18.96.193:39715 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":57,"threads_started":1}
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:37.607071 28693 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:37.602066 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:43063)
0501 02:03:37.602436 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:37.602451 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:37.602575 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:37.602883 (+ 308us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:37.602886 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:37.603069 (+ 183us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:37.603298 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.603303 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.604105 (+ 802us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.604108 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:37.604505 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.604511 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.604604 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.605166 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:37.605180 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:37.606960 (+ 1780us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
W20260501 02:03:37.607209 28692 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:37.602092 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:43063 (local address 127.18.96.254:38985)
0501 02:03:37.602269 (+ 177us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:37.602277 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:37.602544 (+ 267us) server_negotiation.cc:408] Connection header received
0501 02:03:37.602710 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:37.602714 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:37.602769 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:37.602843 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:37.603397 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.604004 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.604599 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.604725 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.607084 (+ 2359us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:43063: BlockingRecv error: recv got EOF from 127.18.96.194:43063 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:37 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:37.775400 28694 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:37.770785 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:44403)
0501 02:03:37.771037 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:37.771050 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:37.771142 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:37.771367 (+ 225us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:37.771369 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:37.771545 (+ 176us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:37.771711 (+ 166us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.771715 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.772373 (+ 658us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.772375 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:37.772709 (+ 334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:37.772714 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.772810 (+ 96us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.773401 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:37.773418 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:37.775297 (+ 1879us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
W20260501 02:03:37.775547 28692 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:37.770868 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:44403 (local address 127.18.96.254:38985)
0501 02:03:37.771020 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:37.771026 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:37.771124 (+ 98us) server_negotiation.cc:408] Connection header received
0501 02:03:37.771224 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:37.771227 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:37.771272 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:37.771336 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:37.771795 (+ 459us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.772288 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:37.772825 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:37.773037 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:37.775406 (+ 2369us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:44403: BlockingRecv error: recv got EOF from 127.18.96.195:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:38.446825 28695 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:38.441885 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:43967)
0501 02:03:38.442153 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:38.442166 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:38.442257 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:38.442543 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:38.442546 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:38.442744 (+ 198us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:38.442945 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.442952 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.443654 (+ 702us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.443657 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:38.444044 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.444050 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.444138 (+ 88us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.444672 (+ 534us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:38.444685 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:38.446709 (+ 2024us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
W20260501 02:03:38.446956 28696 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:38.441943 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:43967 (local address 127.18.96.254:38985)
0501 02:03:38.442211 (+ 268us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:38.442214 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:38.442226 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:03:38.442337 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:38.442339 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:38.442400 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:38.442476 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:38.443061 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.443564 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.444137 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.444264 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.446833 (+ 2569us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:43967: BlockingRecv error: recv got EOF from 127.18.96.193:43967 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":56,"threads_started":1}
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:38.612443 28697 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:38.607830 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:43433)
0501 02:03:38.608162 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:38.608184 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:38.608283 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:38.608502 (+ 219us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:38.608504 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:38.608686 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:38.608871 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.608875 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.609546 (+ 671us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.609549 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:38.609923 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.609927 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.610091 (+ 164us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.610646 (+ 555us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:38.610659 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:38.612328 (+ 1669us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
W20260501 02:03:38.612535 28696 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:38.608091 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:43433 (local address 127.18.96.254:38985)
0501 02:03:38.608217 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:38.608220 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:38.608233 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:38.608360 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:38.608362 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:38.608405 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:38.608469 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:38.608962 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.609441 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.610089 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.610214 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.612440 (+ 2226us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:43433: BlockingRecv error: recv got EOF from 127.18.96.194:43433 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:38 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:38.780952 28698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:38.776244 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:35913)
0501 02:03:38.776519 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:38.776532 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:38.776657 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:38.776911 (+ 254us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:38.776914 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:38.777146 (+ 232us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:38.777330 (+ 184us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.777336 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.778074 (+ 738us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.778079 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:38.778452 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:38.778458 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.778586 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.779117 (+ 531us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:38.779129 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:38.780837 (+ 1708us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
W20260501 02:03:38.781078 28696 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:38.776317 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:35913 (local address 127.18.96.254:38985)
0501 02:03:38.776476 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:38.776481 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:38.776603 (+ 122us) server_negotiation.cc:408] Connection header received
0501 02:03:38.776751 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:38.776756 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:38.776807 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:38.776890 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:38.777456 (+ 566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.777928 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:38.778564 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:38.778776 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:38.780960 (+ 2184us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:35913: BlockingRecv error: recv got EOF from 127.18.96.195:35913 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:39.452409 28699 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:39.447762 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:41115)
0501 02:03:39.448049 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:39.448061 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:39.448199 (+ 138us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:39.448490 (+ 291us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:39.448493 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:39.448665 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:39.448861 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.448865 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.449550 (+ 685us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.449552 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:39.449913 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.449918 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.450073 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.450552 (+ 479us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:39.450564 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:39.452308 (+ 1744us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
W20260501 02:03:39.452569 28700 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:39.447793 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:41115 (local address 127.18.96.254:38985)
0501 02:03:39.448076 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:39.448080 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:39.448161 (+ 81us) server_negotiation.cc:408] Connection header received
0501 02:03:39.448310 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:39.448314 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:39.448369 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:39.448462 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:39.448957 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.449460 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.450095 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.450258 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.452416 (+ 2158us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:41115: BlockingRecv error: recv got EOF from 127.18.96.193:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:39.618130 28701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:39.613349 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:43681)
0501 02:03:39.613642 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:39.613654 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:39.613745 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:39.613990 (+ 245us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:39.613992 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:39.614182 (+ 190us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:39.614386 (+ 204us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.614391 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.615078 (+ 687us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.615081 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:39.615461 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.615467 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.615557 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.616097 (+ 540us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:39.616110 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:39.617979 (+ 1869us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":127,"threads_started":1}
W20260501 02:03:39.618448 28700 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:39.613440 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:43681 (local address 127.18.96.254:38985)
0501 02:03:39.613585 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:39.613588 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:39.613717 (+ 129us) server_negotiation.cc:408] Connection header received
0501 02:03:39.613818 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:39.613820 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:39.613867 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:39.613935 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:39.614491 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.614970 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.615557 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.615686 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.618311 (+ 2625us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:43681: BlockingRecv error: recv got EOF from 127.18.96.194:43681 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:39 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:39.786650 28702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:39.781865 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:52563)
0501 02:03:39.782160 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:39.782174 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:39.782308 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:39.782571 (+ 263us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:39.782574 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:39.782770 (+ 196us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:39.782968 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.782973 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.783711 (+ 738us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.783716 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:39.784114 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:39.784120 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.784244 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.784742 (+ 498us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:39.784754 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:39.786543 (+ 1789us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
W20260501 02:03:39.786777 28700 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:39.781927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:52563 (local address 127.18.96.254:38985)
0501 02:03:39.782113 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:39.782117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:39.782250 (+ 133us) server_negotiation.cc:408] Connection header received
0501 02:03:39.782411 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:39.782415 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:39.782468 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:39.782551 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:39.783107 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.783606 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:39.784226 (+ 620us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:39.784431 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:39.786666 (+ 2235us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:52563: BlockingRecv error: recv got EOF from 127.18.96.195:52563 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:40.462590 28703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:40.453253 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:50265)
0501 02:03:40.453549 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:40.453561 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:40.453652 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:40.453896 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:40.453898 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:40.454092 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:40.454282 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.454288 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.455199 (+ 911us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.455204 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:40.455595 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.455601 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.455761 (+ 160us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.460581 (+ 4820us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:40.460594 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:40.462420 (+ 1826us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":66,"threads_started":1}
W20260501 02:03:40.462647 28704 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:40.453338 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:50265 (local address 127.18.96.254:38985)
0501 02:03:40.453607 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:40.453611 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:40.453624 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:40.453739 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:40.453741 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:40.453784 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:40.453860 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:40.454398 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.455036 (+ 638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.455743 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.455933 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.462537 (+ 6604us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:50265: BlockingRecv error: recv got EOF from 127.18.96.193:50265 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:40.623385 28705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:40.618859 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:50729)
0501 02:03:40.619159 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:40.619173 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:40.619271 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:40.619515 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:40.619517 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:40.619690 (+ 173us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:40.619881 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.619886 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.620567 (+ 681us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.620569 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:40.620934 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.620940 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.621024 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.621521 (+ 497us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:40.621533 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:40.623262 (+ 1729us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
W20260501 02:03:40.623545 28704 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:40.618944 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:50729 (local address 127.18.96.254:38985)
0501 02:03:40.619104 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:40.619109 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:40.619242 (+ 133us) server_negotiation.cc:408] Connection header received
0501 02:03:40.619360 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:40.619363 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:40.619412 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:40.619482 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:40.619978 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.620472 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.621024 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.621149 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.623391 (+ 2242us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:50729: BlockingRecv error: recv got EOF from 127.18.96.194:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:40 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:40.792507 28706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:40.787557 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:43473)
0501 02:03:40.787846 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:40.787860 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:40.787959 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:40.788189 (+ 230us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:40.788192 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:40.788374 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:40.788561 (+ 187us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.788566 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.789321 (+ 755us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.789325 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:40.789874 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:40.789884 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.790065 (+ 181us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.790659 (+ 594us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:40.790676 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:40.792397 (+ 1721us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":88,"threads_started":1}
W20260501 02:03:40.792730 28704 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:40.787633 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:43473 (local address 127.18.96.254:38985)
0501 02:03:40.787767 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:40.787770 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:40.787926 (+ 156us) server_negotiation.cc:408] Connection header received
0501 02:03:40.788034 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:40.788036 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:40.788084 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:40.788153 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:40.788656 (+ 503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.789196 (+ 540us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:40.790070 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:40.790248 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:40.792507 (+ 2259us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:43473: BlockingRecv error: recv got EOF from 127.18.96.195:43473 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:41.468581 28707 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:41.463478 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:39479)
0501 02:03:41.463790 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:41.463803 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:41.463922 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:41.464187 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:41.464191 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:41.464388 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:41.464651 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.464657 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.465458 (+ 801us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.465464 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:41.465863 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.465869 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.466024 (+ 155us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.466576 (+ 552us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:41.466589 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:41.468443 (+ 1854us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
W20260501 02:03:41.468775 28708 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:41.463589 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39479 (local address 127.18.96.254:38985)
0501 02:03:41.463850 (+ 261us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:41.463853 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:41.463877 (+ 24us) server_negotiation.cc:408] Connection header received
0501 02:03:41.464003 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:41.464005 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:41.464064 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:41.464167 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:41.464764 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.465357 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.465992 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.466203 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.468651 (+ 2448us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:39479: BlockingRecv error: recv got EOF from 127.18.96.193:39479 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:41.629402 28709 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:41.624246 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:59351)
0501 02:03:41.624537 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:41.624573 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:41.624694 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:41.624999 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:41.625003 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:41.625249 (+ 246us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:41.625496 (+ 247us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.625502 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.626380 (+ 878us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.626383 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:41.626836 (+ 453us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.626843 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.626936 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.627475 (+ 539us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:41.627487 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:41.629250 (+ 1763us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
W20260501 02:03:41.629455 28708 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:41.624316 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:59351 (local address 127.18.96.254:38985)
0501 02:03:41.624469 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:41.624473 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:41.624675 (+ 202us) server_negotiation.cc:408] Connection header received
0501 02:03:41.624797 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:41.624800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:41.624873 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:41.624946 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:41.625603 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.626266 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.626957 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.627167 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.629361 (+ 2194us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:59351: BlockingRecv error: recv got EOF from 127.18.96.194:59351 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:41 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:41.798460 28710 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:41.793236 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:44693)
0501 02:03:41.793578 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:41.793591 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:41.793726 (+ 135us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:41.794010 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:41.794013 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:41.794213 (+ 200us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:41.794443 (+ 230us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.794451 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.795413 (+ 962us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.795416 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:41.795898 (+ 482us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:41.795905 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.796000 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.796549 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:41.796563 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:41.798357 (+ 1794us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
W20260501 02:03:41.798630 28708 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:41.793381 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:44693 (local address 127.18.96.254:38985)
0501 02:03:41.793535 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:41.793539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:41.793664 (+ 125us) server_negotiation.cc:408] Connection header received
0501 02:03:41.793820 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:41.793826 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:41.793880 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:41.793967 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:41.794574 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.795282 (+ 708us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:41.796007 (+ 725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:41.796155 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:41.798461 (+ 2306us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:44693: BlockingRecv error: recv got EOF from 127.18.96.195:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:42.474829 28711 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:42.469364 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:59413)
0501 02:03:42.469636 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:42.469650 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:42.469741 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:42.469984 (+ 243us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:42.469986 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:42.470176 (+ 190us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:42.470378 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.470384 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.471081 (+ 697us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.471083 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:42.471658 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.471667 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.471780 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.472449 (+ 669us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:42.472471 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:42.474723 (+ 2252us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
W20260501 02:03:42.474974 28712 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:42.469447 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:59413 (local address 127.18.96.254:38985)
0501 02:03:42.469693 (+ 246us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:42.469697 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:42.469709 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:03:42.469819 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:42.469822 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:42.469864 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:42.469945 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:42.470480 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.470974 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.471808 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.472019 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.474847 (+ 2828us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:59413: BlockingRecv error: recv got EOF from 127.18.96.193:59413 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:42.634565 28713 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:42.630152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:45169)
0501 02:03:42.630428 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:42.630440 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:42.630532 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:42.630745 (+ 213us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:42.630747 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:42.630911 (+ 164us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:42.631103 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.631108 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.631754 (+ 646us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.631757 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:42.632133 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.632138 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.632222 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.632705 (+ 483us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:42.632717 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:42.634457 (+ 1740us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":120,"threads_started":1}
W20260501 02:03:42.634709 28712 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:42.630204 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:45169 (local address 127.18.96.254:38985)
0501 02:03:42.630378 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:42.630384 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:42.630506 (+ 122us) server_negotiation.cc:408] Connection header received
0501 02:03:42.630605 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:42.630607 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:42.630651 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:42.630716 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:42.631194 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.631650 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.632223 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.632361 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.634566 (+ 2205us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:45169: BlockingRecv error: recv got EOF from 127.18.96.194:45169 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:42 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:42.804008 28714 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:42.799223 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:54877)
0501 02:03:42.799494 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:42.799507 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:42.799616 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:42.799841 (+ 225us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:42.799843 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:42.800009 (+ 166us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:42.800245 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.800260 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.800928 (+ 668us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.800931 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:42.801293 (+ 362us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:42.801298 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.801403 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.801874 (+ 471us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:42.801886 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:42.803839 (+ 1953us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
W20260501 02:03:42.804044 28712 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:42.799276 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:54877 (local address 127.18.96.254:38985)
0501 02:03:42.799426 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:42.799430 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:42.799579 (+ 149us) server_negotiation.cc:408] Connection header received
0501 02:03:42.799693 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:42.799696 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:42.799741 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:42.799810 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:42.800359 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.800835 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:42.801390 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:42.801510 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:42.803951 (+ 2441us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:54877: BlockingRecv error: recv got EOF from 127.18.96.195:54877 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:43.481432 28715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:43.475613 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:57079)
0501 02:03:43.475928 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:43.475943 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:43.476085 (+ 142us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:43.476407 (+ 322us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:43.476411 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:43.476616 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:43.476888 (+ 272us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.476896 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.478086 (+ 1190us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.478090 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:43.478687 (+ 597us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.478695 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.478817 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.479315 (+ 498us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:43.479332 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:43.481318 (+ 1986us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
W20260501 02:03:43.481601 28716 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:43.475683 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57079 (local address 127.18.96.254:38985)
0501 02:03:43.476014 (+ 331us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:43.476018 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:43.476034 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:03:43.476206 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:43.476209 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:43.476263 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:43.476395 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:43.477023 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.477916 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.478812 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.479024 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.481441 (+ 2417us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:57079: BlockingRecv error: recv got EOF from 127.18.96.193:57079 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:43.640550 28717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:43.635300 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:34467)
0501 02:03:43.635628 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:43.635645 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:43.635785 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:43.636041 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:43.636045 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:43.636247 (+ 202us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:43.636466 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.636472 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.637206 (+ 734us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.637213 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:43.637610 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.637616 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.637742 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.638430 (+ 688us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:43.638442 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:43.640443 (+ 2001us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
W20260501 02:03:43.640704 28716 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:43.635488 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34467 (local address 127.18.96.254:38985)
0501 02:03:43.635628 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:43.635634 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:43.635728 (+ 94us) server_negotiation.cc:408] Connection header received
0501 02:03:43.635879 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:43.635883 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:43.635937 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:43.636022 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:43.636592 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.637087 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.637727 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.637928 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.640562 (+ 2634us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:34467: BlockingRecv error: recv got EOF from 127.18.96.194:34467 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:43 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:43.810024 28718 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:43.804753 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:44169)
0501 02:03:43.805020 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:43.805033 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:43.805139 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:43.805411 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:43.805414 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:43.805607 (+ 193us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:43.805801 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.805806 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.806587 (+ 781us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.806590 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:43.806969 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:43.806975 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.807105 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.807587 (+ 482us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:43.807599 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:43.809855 (+ 2256us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
W20260501 02:03:43.810118 28716 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:43.804888 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:44169 (local address 127.18.96.254:38985)
0501 02:03:43.805020 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:43.805023 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:43.805100 (+ 77us) server_negotiation.cc:408] Connection header received
0501 02:03:43.805214 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:43.805216 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:43.805260 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:43.805328 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:43.805921 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.806489 (+ 568us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:43.807062 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:43.807185 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:43.809998 (+ 2813us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:44169: BlockingRecv error: recv got EOF from 127.18.96.195:44169 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:44.487310 28719 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:44.482223 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:37213)
0501 02:03:44.482638 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:44.482653 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:44.482778 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:44.483065 (+ 287us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:44.483068 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:44.483308 (+ 240us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:44.483584 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.483593 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.484353 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.484357 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:44.484753 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.484758 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.484881 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.485420 (+ 539us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:44.485433 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:44.487203 (+ 1770us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
W20260501 02:03:44.487478 28720 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:44.482283 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:37213 (local address 127.18.96.254:38985)
0501 02:03:44.482527 (+ 244us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:44.482531 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:44.482723 (+ 192us) server_negotiation.cc:408] Connection header received
0501 02:03:44.482890 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:44.482896 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:44.482957 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:44.483046 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:44.483699 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.484235 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.484865 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.485070 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.487319 (+ 2249us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:37213: BlockingRecv error: recv got EOF from 127.18.96.193:37213 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":62,"threads_started":1}
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:44.646270 28721 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:44.641456 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:39947)
0501 02:03:44.641728 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:44.641741 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:44.641833 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:44.642161 (+ 328us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:44.642164 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:44.642373 (+ 209us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:44.642556 (+ 183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.642562 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.643273 (+ 711us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.643278 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:44.643654 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.643660 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.643783 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.644300 (+ 517us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:44.644312 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:44.646139 (+ 1827us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
W20260501 02:03:44.646358 28720 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:44.641550 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:39947 (local address 127.18.96.254:38985)
0501 02:03:44.641683 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:44.641687 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:44.641816 (+ 129us) server_negotiation.cc:408] Connection header received
0501 02:03:44.641969 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:44.641971 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:44.642016 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:44.642140 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:44.642679 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.643163 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.643767 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.643983 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.646269 (+ 2286us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:39947: BlockingRecv error: recv got EOF from 127.18.96.194:39947 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:44 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:44.815618 28722 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:44.810752 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:52415)
0501 02:03:44.811032 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:44.811045 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:44.811157 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:44.811412 (+ 255us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:44.811414 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:44.811621 (+ 207us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:44.811806 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.811811 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.812541 (+ 730us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.812544 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:44.812906 (+ 362us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:44.812911 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.812994 (+ 83us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.813554 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:44.813568 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:44.815511 (+ 1943us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
W20260501 02:03:44.815842 28720 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:44.810895 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:52415 (local address 127.18.96.254:38985)
0501 02:03:44.811048 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:44.811051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:44.811098 (+ 47us) server_negotiation.cc:408] Connection header received
0501 02:03:44.811245 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:44.811248 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:44.811297 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:44.811368 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:44.811921 (+ 553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.812443 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:44.813015 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:44.813226 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:44.815626 (+ 2400us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:52415: BlockingRecv error: recv got EOF from 127.18.96.195:52415 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:45.496589 28723 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:45.488044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:42611)
0501 02:03:45.488397 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:45.488409 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:45.488500 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:45.488710 (+ 210us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:45.488712 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:45.488871 (+ 159us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:45.489050 (+ 179us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.489055 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.489713 (+ 658us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.489715 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:45.490121 (+ 406us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.490126 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.490211 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.494631 (+ 4420us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:45.494645 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:45.496460 (+ 1815us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
W20260501 02:03:45.496753 28724 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:45.488091 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:42611 (local address 127.18.96.254:38985)
0501 02:03:45.488348 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:45.488351 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:45.488470 (+ 119us) server_negotiation.cc:408] Connection header received
0501 02:03:45.488565 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:45.488567 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:45.488608 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:45.488680 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:45.489138 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.489622 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.490209 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.490342 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.496596 (+ 6254us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:42611: BlockingRecv error: recv got EOF from 127.18.96.193:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":93,"threads_started":1}
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:45.651831 28725 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:45.647084 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:55277)
0501 02:03:45.647404 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:45.647417 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:45.647545 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:45.647866 (+ 321us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:45.647869 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:45.648123 (+ 254us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:45.648317 (+ 194us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.648323 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.649015 (+ 692us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.649017 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:45.649420 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.649426 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.649511 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.650013 (+ 502us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:45.650025 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:45.651728 (+ 1703us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260501 02:03:45.651945 28724 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:45.647190 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55277 (local address 127.18.96.254:38985)
0501 02:03:45.647343 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:45.647349 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:45.647473 (+ 124us) server_negotiation.cc:408] Connection header received
0501 02:03:45.647687 (+ 214us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:45.647692 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:45.647748 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:45.647825 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:45.648421 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.648925 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.649512 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.649642 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.651839 (+ 2197us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:55277: BlockingRecv error: recv got EOF from 127.18.96.194:55277 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:45 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:45.821375 28726 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:45.816415 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:41041)
0501 02:03:45.816684 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:45.816697 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:45.816800 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:45.817036 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:45.817039 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:45.817215 (+ 176us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:45.817423 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.817429 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.818293 (+ 864us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.818297 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:45.818775 (+ 478us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:45.818784 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.818890 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.819452 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:45.819464 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:45.821225 (+ 1761us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
W20260501 02:03:45.821480 28724 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:45.816498 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:41041 (local address 127.18.96.254:38985)
0501 02:03:45.816682 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:45.816685 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:45.816771 (+ 86us) server_negotiation.cc:408] Connection header received
0501 02:03:45.816883 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:45.816886 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:45.816932 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:45.817001 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:45.817540 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.818175 (+ 635us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:45.818891 (+ 716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:45.819034 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:45.821355 (+ 2321us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:41041: BlockingRecv error: recv got EOF from 127.18.96.195:41041 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:46.501782 28727 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:46.497368 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:58935)
0501 02:03:46.497628 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:46.497640 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:46.497731 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:46.498020 (+ 289us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:46.498022 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:46.498181 (+ 159us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:46.498367 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.498372 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.498998 (+ 626us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.499000 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:46.499338 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.499342 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.499455 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.499977 (+ 522us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:46.499990 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:46.501659 (+ 1669us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
W20260501 02:03:46.501912 28728 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:46.497427 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:58935 (local address 127.18.96.254:38985)
0501 02:03:46.497666 (+ 239us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:46.497669 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:46.497685 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:03:46.497823 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:46.497825 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:46.497868 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:46.497938 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:46.498460 (+ 522us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.498913 (+ 453us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.499453 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.499573 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.501792 (+ 2219us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:58935: BlockingRecv error: recv got EOF from 127.18.96.193:58935 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":60,"threads_started":1}
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:46.657661 28729 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:46.652652 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:52525)
0501 02:03:46.652976 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:46.652989 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:46.653082 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:46.653359 (+ 277us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:46.653361 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:46.653605 (+ 244us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:46.653818 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.653823 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.654595 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.654598 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:46.655031 (+ 433us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.655038 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.655151 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.655742 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:46.655754 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:46.657551 (+ 1797us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":74,"threads_started":1}
W20260501 02:03:46.657780 28728 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:46.652766 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:52525 (local address 127.18.96.254:38985)
0501 02:03:46.652882 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:46.652890 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:46.653056 (+ 166us) server_negotiation.cc:408] Connection header received
0501 02:03:46.653169 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:46.653191 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:46.653259 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:46.653343 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:46.653948 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.654499 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.655143 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.655274 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.657667 (+ 2393us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:52525: BlockingRecv error: recv got EOF from 127.18.96.194:52525 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:46 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:46.827438 28730 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:46.822269 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:49399)
0501 02:03:46.822567 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:46.822583 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:46.822711 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:46.822993 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:46.822999 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:46.823250 (+ 251us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:46.823523 (+ 273us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.823532 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.824290 (+ 758us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.824293 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:46.824663 (+ 370us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:46.824668 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.824752 (+ 84us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.825301 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:46.825314 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:46.827290 (+ 1976us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
W20260501 02:03:46.827517 28728 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:46.822342 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:49399 (local address 127.18.96.254:38985)
0501 02:03:46.822492 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:46.822497 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:46.822648 (+ 151us) server_negotiation.cc:408] Connection header received
0501 02:03:46.822814 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:46.822818 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:46.822878 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:46.822945 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:46.823651 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.824198 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:46.824755 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:46.824881 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:46.827392 (+ 2511us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:49399: BlockingRecv error: recv got EOF from 127.18.96.195:49399 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:47.507462 28731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:47.502540 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:54855)
0501 02:03:47.502834 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:47.502848 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:47.502934 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:47.503205 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:47.503207 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:47.503421 (+ 214us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:47.503606 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.503611 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.504402 (+ 791us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.504404 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:47.504777 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.504782 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.504871 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.505449 (+ 578us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:47.505463 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:47.507328 (+ 1865us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
W20260501 02:03:47.507654 28732 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:47.502660 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:54855 (local address 127.18.96.254:38985)
0501 02:03:47.502933 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:47.502936 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:47.502948 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:03:47.503001 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:47.503003 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:47.503044 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:47.503168 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:47.503704 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.504294 (+ 590us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.504868 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.504993 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.507494 (+ 2501us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:54855: BlockingRecv error: recv got EOF from 127.18.96.193:54855 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:47.663291 28733 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:47.658546 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:58473)
0501 02:03:47.658832 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:47.658850 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:47.658986 (+ 136us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:47.659270 (+ 284us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:47.659273 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:47.659498 (+ 225us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:47.659680 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.659685 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.660420 (+ 735us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.660424 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:47.660793 (+ 369us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.660799 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.660929 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.661418 (+ 489us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:47.661430 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:47.663116 (+ 1686us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
W20260501 02:03:47.663357 28732 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:47.658630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:58473 (local address 127.18.96.254:38985)
0501 02:03:47.658754 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:47.658758 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:47.658940 (+ 182us) server_negotiation.cc:408] Connection header received
0501 02:03:47.659081 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:47.659084 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:47.659145 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:47.659238 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:47.659801 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.660306 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.660923 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.661131 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.663236 (+ 2105us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:58473: BlockingRecv error: recv got EOF from 127.18.96.194:58473 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:47 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.195@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:47.833155 28734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:47.828209 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.195:39069)
0501 02:03:47.828505 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:47.828517 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:47.828639 (+ 122us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:47.828947 (+ 308us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:47.828950 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:47.829225 (+ 275us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:47.829457 (+ 232us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.829464 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.830258 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.830264 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:47.830666 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:47.830672 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.830799 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.831306 (+ 507us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:47.831319 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:47.833027 (+ 1708us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
W20260501 02:03:47.833391 28732 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:47.828267 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:39069 (local address 127.18.96.254:38985)
0501 02:03:47.828425 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:47.828429 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:47.828584 (+ 155us) server_negotiation.cc:408] Connection header received
0501 02:03:47.828792 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:47.828794 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:47.828840 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:47.828925 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:47.829573 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.830150 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:47.830784 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:47.830989 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:47.833164 (+ 2175us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.195:39069: BlockingRecv error: recv got EOF from 127.18.96.195:39069 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
May 01 02:03:48 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:48 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.193@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:48.512831 28735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:48.508376 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.193:40127)
0501 02:03:48.508649 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:48.508663 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:48.508749 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:48.508993 (+ 244us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:48.508995 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:48.509162 (+ 167us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:48.509344 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:48.509348 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.510080 (+ 732us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:48.510083 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:48.510451 (+ 368us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:48.510456 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.510534 (+ 78us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:48.510980 (+ 446us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:48.510991 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:48.512719 (+ 1728us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
W20260501 02:03:48.512945 28736 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:48.508507 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:40127 (local address 127.18.96.254:38985)
0501 02:03:48.508725 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:48.508728 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:48.508741 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:03:48.508838 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:48.508840 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:48.508883 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:48.508958 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:48.509463 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.509942 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:48.510536 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.510658 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:48.512832 (+ 2174us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.193:40127: BlockingRecv error: recv got EOF from 127.18.96.193:40127 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":57,"threads_started":1}
May 01 02:03:48 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
May 01 02:03:48 dist-test-slave-2j90 krb5kdc[27669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.96.194@KRBTEST.COM for oryx/127.18.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260501 02:03:48.668736 28737 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:48.664258 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38985 (local address 127.18.96.194:59169)
0501 02:03:48.664523 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:48.664536 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:48.664630 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:48.664856 (+ 226us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:48.664858 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:48.665028 (+ 170us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:48.665218 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:48.665223 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.665905 (+ 682us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:48.665907 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:48.666341 (+ 434us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:48.666348 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.666446 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:48.666987 (+ 541us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:48.667000 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:48.668635 (+ 1635us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.96.254:38985: Server oryx/127.18.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
W20260501 02:03:48.668879 28736 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:48.664347 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:59169 (local address 127.18.96.254:38985)
0501 02:03:48.664523 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:48.664526 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:48.664606 (+ 80us) server_negotiation.cc:408] Connection header received
0501 02:03:48.664707 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:48.664709 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:48.664753 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:48.664822 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:48.665317 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.665815 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:48.666448 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.666578 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:48.668742 (+ 2164us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.96.194:59169: BlockingRecv error: recv got EOF from 127.18.96.194:59169 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260501 02:03:48.740772 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28261
I20260501 02:03:48.746411 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28413
I20260501 02:03:48.751870 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28545
I20260501 02:03:48.757380 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28191
2026-05-01T02:03:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18750 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-05-01T02:03:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:48Z Disabled control of system clock
I20260501 02:03:48.786473 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39451
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260501 02:03:48.883024 28741 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:48.883208 28741 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:48.883247 28741 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:48.886430 28741 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:48.886477 28741 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:48.886494 28741 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:48.886512 28741 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:48.886529 28741 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:48.890214 28741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:39451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:48.891098 28741 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:48.892063 28741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:48.897588 28749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:48.897588 28746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:48.897843 28747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:48.897989 28741 server_base.cc:1061] running on GCE node
I20260501 02:03:48.898407 28741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:48.899273 28741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:48.900463 28741 hybrid_clock.cc:648] HybridClock initialized: now 1777601028900444 us; error 38 us; skew 500 ppm
I20260501 02:03:48.902134 28741 webserver.cc:492] Webserver started at http://127.18.96.254:45123/ using document root <none> and password file <none>
I20260501 02:03:48.902645 28741 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:48.902725 28741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:48.902949 28741 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:48.904510 28741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "0b663b10786b4961b73076b5d69a1139"
format_stamp: "Formatted at 2026-05-01 02:03:48 on dist-test-slave-2j90"
server_key: "6c548679c6c6b27df64ff8b8f91a1ae4"
server_key_iv: "c6aab470b5d6e7c261ddb40a5e457ecc"
server_key_version: "encryptionkey@0"
I20260501 02:03:48.904982 28741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "0b663b10786b4961b73076b5d69a1139"
format_stamp: "Formatted at 2026-05-01 02:03:48 on dist-test-slave-2j90"
server_key: "6c548679c6c6b27df64ff8b8f91a1ae4"
server_key_iv: "c6aab470b5d6e7c261ddb40a5e457ecc"
server_key_version: "encryptionkey@0"
I20260501 02:03:48.907972 28741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260501 02:03:48.909881 28755 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:48.910775 28741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:48.910908 28741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "0b663b10786b4961b73076b5d69a1139"
format_stamp: "Formatted at 2026-05-01 02:03:48 on dist-test-slave-2j90"
server_key: "6c548679c6c6b27df64ff8b8f91a1ae4"
server_key_iv: "c6aab470b5d6e7c261ddb40a5e457ecc"
server_key_version: "encryptionkey@0"
I20260501 02:03:48.910997 28741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:48.925426 28741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:48.926033 28741 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:48.926211 28741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:48.932780 28741 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:39451
I20260501 02:03:48.932792 28807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:39451 every 8 connection(s)
I20260501 02:03:48.933679 28741 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:48.936321 28808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:48.940850 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28741
I20260501 02:03:48.940982 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:48.941234 18819 external_mini_cluster.cc:1468] Setting key 467eac53ecec9857dc65d292d33030ce
I20260501 02:03:48.941323 28808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139: Bootstrap starting.
I20260501 02:03:48.943572 28808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:48.944254 28808 log.cc:826] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:48.945760 28808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139: No bootstrap required, opened a new log
I20260501 02:03:48.948175 28811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:48.942413 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.18.96.254:39451)
0501 02:03:48.942955 (+ 542us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:48.942963 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:48.942993 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:03:48.943465 (+ 472us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:48.943478 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:48.943820 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:48.944112 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:48.944608 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.945322 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:48.946080 (+ 758us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:48.946366 (+ 286us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:48.946878 (+ 512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:48.946901 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:48.946910 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:48.947231 (+ 321us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:48.947262 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:48.947274 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:48.947397 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:48.947562 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:03:48.947693 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":141,"threads_started":1}
I20260501 02:03:48.948343 28808 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } }
I20260501 02:03:48.948539 28808 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:48.948614 28808 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b663b10786b4961b73076b5d69a1139, State: Initialized, Role: FOLLOWER
I20260501 02:03:48.949029 28808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } }
I20260501 02:03:48.949177 28808 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:48.949244 28808 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:48.949361 28808 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:48.950207 28808 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } }
I20260501 02:03:48.950508 28808 leader_election.cc:304] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b663b10786b4961b73076b5d69a1139; no voters:
I20260501 02:03:48.950773 28808 leader_election.cc:290] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:48.950888 28813 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:48.951135 28813 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [term 1 LEADER]: Becoming Leader. State: Replica: 0b663b10786b4961b73076b5d69a1139, State: Running, Role: LEADER
I20260501 02:03:48.951463 28813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } }
I20260501 02:03:48.951792 28808 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:48.952913 28814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b663b10786b4961b73076b5d69a1139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } } }
I20260501 02:03:48.953037 28814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:48.953416 28822 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:48.953640 28815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b663b10786b4961b73076b5d69a1139. Latest consensus state: current_term: 1 leader_uuid: "0b663b10786b4961b73076b5d69a1139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b663b10786b4961b73076b5d69a1139" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39451 } } }
I20260501 02:03:48.953737 28815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:48.956372 28822 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:48.961462 28822 catalog_manager.cc:1357] Generated new cluster ID: 1db0f80303524bffb081935b3ff6b99b
I20260501 02:03:48.961542 28822 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:49.024552 28822 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:49.025482 28822 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:49.032940 28822 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0b663b10786b4961b73076b5d69a1139: Generated new TSK 0
I20260501 02:03:49.033550 28822 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:03:49.038311 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--builtin_ntp_servers=127.18.96.212:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260501 02:03:49.153162 28832 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:49.153436 28832 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:49.153489 28832 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:49.156965 28832 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:49.157018 28832 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:49.157094 28832 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:49.161016 28832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:49.162091 28832 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:49.163116 28832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:49.169322 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.169348 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.169381 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.169559 28832 server_base.cc:1061] running on GCE node
I20260501 02:03:49.169945 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:49.170562 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:49.171710 28832 hybrid_clock.cc:648] HybridClock initialized: now 1777601029171685 us; error 38 us; skew 500 ppm
I20260501 02:03:49.173427 28832 webserver.cc:492] Webserver started at http://127.18.96.193:37651/ using document root <none> and password file <none>
I20260501 02:03:49.174031 28832 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:49.174096 28832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:49.174319 28832 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:49.175892 28832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "8607c8e7cf7e4fca8a6906ae2c7b1288"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "43bfe60494c7d788c3044b9847981570"
server_key_iv: "878186a74f2286c5d51ba7ce22392845"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.176396 28832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "8607c8e7cf7e4fca8a6906ae2c7b1288"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "43bfe60494c7d788c3044b9847981570"
server_key_iv: "878186a74f2286c5d51ba7ce22392845"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.179651 28832 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:49.181685 28846 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.182628 28832 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:03:49.182750 28832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "8607c8e7cf7e4fca8a6906ae2c7b1288"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "43bfe60494c7d788c3044b9847981570"
server_key_iv: "878186a74f2286c5d51ba7ce22392845"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.182852 28832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:49.200214 28832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:49.200742 28832 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:49.200917 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:49.201438 28832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:49.202298 28832 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:49.202378 28832 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.202455 28832 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:49.202500 28832 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.210815 28832 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:45351
I20260501 02:03:49.210860 28959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:45351 every 8 connection(s)
I20260501 02:03:49.211637 28832 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:49.218951 28811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.213179 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:47979 (local address 127.18.96.254:39451)
0501 02:03:49.213335 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.213339 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.214075 (+ 736us) server_negotiation.cc:408] Connection header received
0501 02:03:49.214945 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.214948 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.215009 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.215106 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.215838 (+ 732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.216357 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.216995 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.217182 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.218452 (+ 1270us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:49.218470 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:49.218473 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:49.218492 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:49.218522 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:49.218529 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:49.218605 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:49.218765 (+ 160us) server_negotiation.cc:300] Negotiation successful
0501 02:03:49.218817 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:03:49.219300 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28832
I20260501 02:03:49.219419 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:49.219676 18819 external_mini_cluster.cc:1468] Setting key 6995cc2ebeedfda2e92e61b26db23f5a
I20260501 02:03:49.220139 28962 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.213414 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39451 (local address 127.18.96.193:47979)
0501 02:03:49.213907 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:49.213935 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:49.214743 (+ 808us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:49.215248 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:49.215255 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:49.215306 (+ 51us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:49.215702 (+ 396us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.215712 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.216490 (+ 778us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.216498 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:49.216878 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.216884 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.217098 (+ 214us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.218231 (+ 1133us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:49.218232 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:49.218254 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:49.218293 (+ 39us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:49.218619 (+ 326us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:49.218628 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:49.218635 (+ 7us) client_negotiation.cc:770] Sending connection context
0501 02:03:49.218766 (+ 131us) client_negotiation.cc:241] Negotiation successful
0501 02:03:49.218883 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
I20260501 02:03:49.221161 28960 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39451
I20260501 02:03:49.221386 28960 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:49.221573 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--builtin_ntp_servers=127.18.96.212:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260501 02:03:49.221930 28960 heartbeater.cc:507] Master 127.18.96.254:39451 requested a full tablet report, sending...
I20260501 02:03:49.223275 28771 ts_manager.cc:194] Registered new tserver with Master: 8607c8e7cf7e4fca8a6906ae2c7b1288 (127.18.96.193:45351)
I20260501 02:03:49.224366 28771 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.193:47979
W20260501 02:03:49.317132 28963 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:49.317337 28963 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:49.317420 28963 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:49.320611 28963 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:49.320688 28963 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:49.320784 28963 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:49.324390 28963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:49.325275 28963 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:49.326277 28963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:49.332104 28969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.332157 28968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.332127 28971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.332365 28963 server_base.cc:1061] running on GCE node
I20260501 02:03:49.332651 28963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:49.333122 28963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:49.334265 28963 hybrid_clock.cc:648] HybridClock initialized: now 1777601029334235 us; error 41 us; skew 500 ppm
I20260501 02:03:49.335763 28963 webserver.cc:492] Webserver started at http://127.18.96.194:44983/ using document root <none> and password file <none>
I20260501 02:03:49.336243 28963 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:49.336292 28963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:49.336527 28963 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:49.338074 28963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "7e2c5be50180449b9161629ce1c4a1c1"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "03be208a42cbaddc28134472b02aa13d"
server_key_iv: "512ba4d54cae4e3247a00ed7ade0fa97"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.338524 28963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "7e2c5be50180449b9161629ce1c4a1c1"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "03be208a42cbaddc28134472b02aa13d"
server_key_iv: "512ba4d54cae4e3247a00ed7ade0fa97"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.341439 28963 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:03:49.343547 28977 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.344451 28963 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:49.344573 28963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "7e2c5be50180449b9161629ce1c4a1c1"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "03be208a42cbaddc28134472b02aa13d"
server_key_iv: "512ba4d54cae4e3247a00ed7ade0fa97"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.344669 28963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:49.362970 28963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:49.363502 28963 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:49.363667 28963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:49.364172 28963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:49.365031 28963 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:49.365103 28963 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.365180 28963 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:49.365231 28963 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.373807 28963 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:33515
I20260501 02:03:49.373883 29090 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:33515 every 8 connection(s)
I20260501 02:03:49.374711 28963 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:49.376674 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 28963
I20260501 02:03:49.376804 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:49.377049 18819 external_mini_cluster.cc:1468] Setting key 29940aa068e187f602396e589a008b17
I20260501 02:03:49.379324 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--builtin_ntp_servers=127.18.96.212:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260501 02:03:49.382663 28811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.376297 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:49845 (local address 127.18.96.254:39451)
0501 02:03:49.376462 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.376466 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.377343 (+ 877us) server_negotiation.cc:408] Connection header received
0501 02:03:49.378344 (+ 1001us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.378347 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.378432 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.378508 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.379428 (+ 920us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.379938 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.380566 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.380740 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.382020 (+ 1280us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:49.382042 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:49.382048 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:49.382064 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:49.382089 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:49.382101 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:49.382173 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:49.382408 (+ 235us) server_negotiation.cc:300] Negotiation successful
0501 02:03:49.382461 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260501 02:03:49.383090 29093 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.376601 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39451 (local address 127.18.96.194:49845)
0501 02:03:49.377200 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:49.377236 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:49.378141 (+ 905us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:49.378746 (+ 605us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:49.378757 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:49.378826 (+ 69us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:49.379284 (+ 458us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.379295 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.380067 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.380070 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:49.380452 (+ 382us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.380458 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.380671 (+ 213us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.381799 (+ 1128us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:49.381801 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:49.381823 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:49.381862 (+ 39us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:49.382221 (+ 359us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:49.382228 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:49.382232 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:03:49.382373 (+ 141us) client_negotiation.cc:241] Negotiation successful
0501 02:03:49.382484 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260501 02:03:49.384202 29091 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39451
I20260501 02:03:49.384449 29091 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:49.384907 29091 heartbeater.cc:507] Master 127.18.96.254:39451 requested a full tablet report, sending...
I20260501 02:03:49.385875 28771 ts_manager.cc:194] Registered new tserver with Master: 7e2c5be50180449b9161629ce1c4a1c1 (127.18.96.194:33515)
I20260501 02:03:49.386508 28771 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.194:49845
W20260501 02:03:49.477416 29094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:49.477612 29094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:49.477677 29094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:49.480849 29094 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:49.480919 29094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:49.481014 29094 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:49.484549 29094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:49.485514 29094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:49.486527 29094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:49.492379 29099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.492444 29100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.492463 29094 server_base.cc:1061] running on GCE node
W20260501 02:03:49.492588 29102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.492875 29094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:49.493331 29094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:49.494473 29094 hybrid_clock.cc:648] HybridClock initialized: now 1777601029494450 us; error 34 us; skew 500 ppm
I20260501 02:03:49.496055 29094 webserver.cc:492] Webserver started at http://127.18.96.195:35331/ using document root <none> and password file <none>
I20260501 02:03:49.496747 29094 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:49.496824 29094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:49.497025 29094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:49.499189 29094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "146f76b355274f2d8093cc3de5f01a2f"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "17de94a2b696e494a523e270e18a0a94"
server_key_iv: "3afa2752b49ec010373b2e354d4a1bc9"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.499779 29094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "146f76b355274f2d8093cc3de5f01a2f"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "17de94a2b696e494a523e270e18a0a94"
server_key_iv: "3afa2752b49ec010373b2e354d4a1bc9"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.502719 29094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:03:49.504629 29108 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.505553 29094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:03:49.505640 29094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "146f76b355274f2d8093cc3de5f01a2f"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "17de94a2b696e494a523e270e18a0a94"
server_key_iv: "3afa2752b49ec010373b2e354d4a1bc9"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.505710 29094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:49.519667 29094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:49.520139 29094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:49.520267 29094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:49.520674 29094 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:49.521517 29094 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:49.521579 29094 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.521615 29094 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:49.521636 29094 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.530201 29094 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:37085
I20260501 02:03:49.530227 29221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:37085 every 8 connection(s)
I20260501 02:03:49.531080 29094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:49.535431 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29094
I20260501 02:03:49.535547 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:49.535799 18819 external_mini_cluster.cc:1468] Setting key 3df4be889cbccebe8f09c85acba020be
I20260501 02:03:49.538856 28811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.532762 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:55375 (local address 127.18.96.254:39451)
0501 02:03:49.532904 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.532908 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.533676 (+ 768us) server_negotiation.cc:408] Connection header received
0501 02:03:49.534569 (+ 893us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.534572 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.534633 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.534748 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.535525 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.536151 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.536990 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.537165 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.538448 (+ 1283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:49.538461 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:49.538463 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:49.538473 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:49.538492 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:49.538498 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:49.538557 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:49.538715 (+ 158us) server_negotiation.cc:300] Negotiation successful
0501 02:03:49.538753 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:03:49.539405 29224 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.532983 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:39451 (local address 127.18.96.195:55375)
0501 02:03:49.533551 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:49.533579 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:49.534384 (+ 805us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:49.534889 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:49.534896 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:49.534953 (+ 57us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:49.535389 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.535399 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.536287 (+ 888us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:49.536290 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:49.536876 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:49.536887 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:49.537100 (+ 213us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:49.538249 (+ 1149us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:49.538250 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:49.538288 (+ 38us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:49.538328 (+ 40us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:49.538594 (+ 266us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:49.538598 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:49.538602 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:03:49.538701 (+ 99us) client_negotiation.cc:241] Negotiation successful
0501 02:03:49.538805 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260501 02:03:49.540284 29222 heartbeater.cc:344] Connected to a master server at 127.18.96.254:39451
I20260501 02:03:49.540493 29222 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:49.540958 29222 heartbeater.cc:507] Master 127.18.96.254:39451 requested a full tablet report, sending...
I20260501 02:03:49.541783 28771 ts_manager.cc:194] Registered new tserver with Master: 146f76b355274f2d8093cc3de5f01a2f (127.18.96.195:37085)
I20260501 02:03:49.542295 28771 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.195:55375
I20260501 02:03:49.549324 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260501 02:03:49.551921 28811 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:49.551026 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.18.96.254:39451)
0501 02:03:49.551181 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.551184 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.551347 (+ 163us) server_negotiation.cc:408] Connection header received
0501 02:03:49.551499 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.551508 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.551545 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.551624 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.551841 (+ 217us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54516: BlockingRecv error: recv got EOF from 127.0.0.1:54516 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:03:49.552217 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28832
I20260501 02:03:49.558292 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28963
I20260501 02:03:49.563410 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29094
I20260501 02:03:49.568852 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 28741
2026-05-01T02:03:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (810 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-05-01T02:03:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:49Z Disabled control of system clock
I20260501 02:03:49.596676 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44919
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260501 02:03:49.691987 29233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:49.692183 29233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:49.692250 29233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:49.695405 29233 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:49.695456 29233 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:49.695473 29233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:49.695492 29233 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:49.695508 29233 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:49.699180 29233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:49.700063 29233 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:49.701014 29233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:49.706248 29238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.706292 29241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.706256 29239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.706425 29233 server_base.cc:1061] running on GCE node
I20260501 02:03:49.707008 29233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:49.707798 29233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:49.708972 29233 hybrid_clock.cc:648] HybridClock initialized: now 1777601029708959 us; error 35 us; skew 500 ppm
I20260501 02:03:49.710632 29233 webserver.cc:492] Webserver started at http://127.18.96.254:36655/ using document root <none> and password file <none>
I20260501 02:03:49.711154 29233 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:49.711233 29233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:49.711468 29233 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:49.712985 29233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "9a7af48411864d03bb003e3bf5fa62e0"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "462d4deaba640ad7c4b368f2b7ce250e"
server_key_iv: "3b66677df8ec4369787cd66a88df4981"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.713451 29233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "9a7af48411864d03bb003e3bf5fa62e0"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "462d4deaba640ad7c4b368f2b7ce250e"
server_key_iv: "3b66677df8ec4369787cd66a88df4981"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.716583 29233 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:49.718592 29247 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.719455 29233 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:49.719581 29233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9a7af48411864d03bb003e3bf5fa62e0"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "462d4deaba640ad7c4b368f2b7ce250e"
server_key_iv: "3b66677df8ec4369787cd66a88df4981"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.719676 29233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:49.724004 29233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:49.724505 29233 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:49.724663 29233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:49.731119 29233 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44919
I20260501 02:03:49.731138 29299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44919 every 8 connection(s)
I20260501 02:03:49.731971 29233 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:49.734447 29300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:49.739225 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0: Bootstrap starting.
I20260501 02:03:49.741281 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29233
I20260501 02:03:49.741307 29300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:49.741396 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:49.741669 18819 external_mini_cluster.cc:1468] Setting key 6c0767c0904e20fdee9942d89de40f24
I20260501 02:03:49.742105 29300 log.cc:826] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:49.743783 29300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0: No bootstrap required, opened a new log
I20260501 02:03:49.746160 29304 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.742678 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.18.96.254:44919)
0501 02:03:49.743181 (+ 503us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.743189 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.743219 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:03:49.743684 (+ 465us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.743697 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.743999 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.744239 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.745128 (+ 889us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:49.745133 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:49.745141 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:49.745151 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:49.745188 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:49.745198 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:49.745285 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:49.745454 (+ 169us) server_negotiation.cc:300] Negotiation successful
0501 02:03:49.745590 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260501 02:03:49.746868 29300 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } }
I20260501 02:03:49.747103 29300 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:49.747169 29300 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7af48411864d03bb003e3bf5fa62e0, State: Initialized, Role: FOLLOWER
I20260501 02:03:49.747591 29300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } }
I20260501 02:03:49.747740 29300 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:49.747807 29300 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:49.747931 29300 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:49.748772 29300 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } }
I20260501 02:03:49.749085 29300 leader_election.cc:304] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a7af48411864d03bb003e3bf5fa62e0; no voters:
I20260501 02:03:49.749351 29300 leader_election.cc:290] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:49.749464 29305 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:49.749665 29305 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a7af48411864d03bb003e3bf5fa62e0, State: Running, Role: LEADER
I20260501 02:03:49.749928 29305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } }
I20260501 02:03:49.750377 29300 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:49.751346 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a7af48411864d03bb003e3bf5fa62e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } } }
I20260501 02:03:49.751482 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:49.751555 29307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a7af48411864d03bb003e3bf5fa62e0. Latest consensus state: current_term: 1 leader_uuid: "9a7af48411864d03bb003e3bf5fa62e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a7af48411864d03bb003e3bf5fa62e0" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44919 } } }
I20260501 02:03:49.751654 29307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:49.752496 29314 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:49.755029 29314 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:49.759789 29314 catalog_manager.cc:1357] Generated new cluster ID: bd2b394e01564d72861f99894dcce8cb
I20260501 02:03:49.759845 29314 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:49.772310 29314 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:49.772535 29314 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:49.778134 29314 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9a7af48411864d03bb003e3bf5fa62e0: Generated new TSK 0
I20260501 02:03:49.778594 29314 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:03:49.785919 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--builtin_ntp_servers=127.18.96.212:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260501 02:03:49.886922 29324 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:49.887137 29324 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:49.887190 29324 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:49.890697 29324 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:49.890744 29324 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:49.890843 29324 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:49.894775 29324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:49.895792 29324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:49.896854 29324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:49.902887 29332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.903012 29330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:49.902899 29329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:49.903149 29324 server_base.cc:1061] running on GCE node
I20260501 02:03:49.903496 29324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:49.904033 29324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:49.905225 29324 hybrid_clock.cc:648] HybridClock initialized: now 1777601029905215 us; error 26 us; skew 500 ppm
I20260501 02:03:49.906876 29324 webserver.cc:492] Webserver started at http://127.18.96.193:38177/ using document root <none> and password file <none>
I20260501 02:03:49.907406 29324 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:49.907490 29324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:49.907698 29324 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:49.909224 29324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "30998418543b41e79c7f0d531291c2cf"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "90b615438240e0856591635b9b6a3531"
server_key_iv: "fdb27f1c1caa39f9f2e78a81c7d11fde"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.909729 29324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "30998418543b41e79c7f0d531291c2cf"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "90b615438240e0856591635b9b6a3531"
server_key_iv: "fdb27f1c1caa39f9f2e78a81c7d11fde"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.912894 29324 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260501 02:03:49.914985 29338 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.915870 29324 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:49.916005 29324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "30998418543b41e79c7f0d531291c2cf"
format_stamp: "Formatted at 2026-05-01 02:03:49 on dist-test-slave-2j90"
server_key: "90b615438240e0856591635b9b6a3531"
server_key_iv: "fdb27f1c1caa39f9f2e78a81c7d11fde"
server_key_version: "encryptionkey@0"
I20260501 02:03:49.916101 29324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:49.920384 29324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:49.920888 29324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:49.921063 29324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:49.921564 29324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:49.922497 29324 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:49.922567 29324 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.922631 29324 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:49.922684 29324 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:49.930920 29324 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:38129
I20260501 02:03:49.930954 29451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:38129 every 8 connection(s)
I20260501 02:03:49.931778 29324 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:49.936642 29304 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.933389 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:34555 (local address 127.18.96.254:44919)
0501 02:03:49.933530 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:49.933534 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:49.934257 (+ 723us) server_negotiation.cc:408] Connection header received
0501 02:03:49.935035 (+ 778us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:49.935039 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:49.935087 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:49.935197 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:49.936165 (+ 968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:49.936169 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:49.936172 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:49.936184 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:49.936206 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:49.936221 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:49.936315 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:49.936471 (+ 156us) server_negotiation.cc:300] Negotiation successful
0501 02:03:49.936507 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:03:49.937238 29454 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:49.933580 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44919 (local address 127.18.96.193:34555)
0501 02:03:49.934129 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:49.934159 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:49.934848 (+ 689us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:49.935336 (+ 488us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:49.935343 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:49.935398 (+ 55us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:49.935951 (+ 553us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:49.935953 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:49.935976 (+ 23us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:49.936018 (+ 42us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:49.936311 (+ 293us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:49.936318 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:49.936325 (+ 7us) client_negotiation.cc:770] Sending connection context
0501 02:03:49.936512 (+ 187us) client_negotiation.cc:241] Negotiation successful
0501 02:03:49.936605 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260501 02:03:49.938403 29452 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44919
I20260501 02:03:49.938612 29452 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:49.938957 29452 heartbeater.cc:507] Master 127.18.96.254:44919 requested a full tablet report, sending...
I20260501 02:03:49.940243 29264 ts_manager.cc:194] Registered new tserver with Master: 30998418543b41e79c7f0d531291c2cf (127.18.96.193:38129)
I20260501 02:03:49.941536 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29324
I20260501 02:03:49.941682 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:49.941944 18819 external_mini_cluster.cc:1468] Setting key ba9c3f69a86acaaf4fbb4971b1401f1b
I20260501 02:03:49.943629 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--builtin_ntp_servers=127.18.96.212:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260501 02:03:50.042300 29455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:50.042488 29455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:50.042534 29455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:50.046240 29455 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:50.046290 29455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:50.046361 29455 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:50.049999 29455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:50.050907 29455 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:50.051978 29455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:50.058049 29461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:50.058096 29460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:50.058096 29463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:50.058282 29455 server_base.cc:1061] running on GCE node
I20260501 02:03:50.058609 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:50.059134 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:50.060297 29455 hybrid_clock.cc:648] HybridClock initialized: now 1777601030060277 us; error 35 us; skew 500 ppm
I20260501 02:03:50.061913 29455 webserver.cc:492] Webserver started at http://127.18.96.194:37911/ using document root <none> and password file <none>
I20260501 02:03:50.062487 29455 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:50.062566 29455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:50.062765 29455 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:50.064306 29455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "9f464c50d5b84104a9b58f0ff24ebc4d"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "6dfb883b26da5e74f4e5af8b3c79cdbf"
server_key_iv: "46fcec0266d37ae07777c5d37640343a"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.064769 29455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "9f464c50d5b84104a9b58f0ff24ebc4d"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "6dfb883b26da5e74f4e5af8b3c79cdbf"
server_key_iv: "46fcec0266d37ae07777c5d37640343a"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.067822 29455 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:50.069851 29469 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.070757 29455 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:50.070881 29455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "9f464c50d5b84104a9b58f0ff24ebc4d"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "6dfb883b26da5e74f4e5af8b3c79cdbf"
server_key_iv: "46fcec0266d37ae07777c5d37640343a"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.070974 29455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:50.075133 29455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:50.075649 29455 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:50.075819 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:50.076292 29455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:50.077158 29455 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:50.077230 29455 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.077296 29455 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:50.077349 29455 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.085862 29455 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:44663
I20260501 02:03:50.085896 29582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:44663 every 8 connection(s)
I20260501 02:03:50.086714 29455 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:50.088464 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29455
I20260501 02:03:50.088596 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:50.088871 18819 external_mini_cluster.cc:1468] Setting key 47d1a2110cf0745edecf85a11653e795
I20260501 02:03:50.090786 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--builtin_ntp_servers=127.18.96.212:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260501 02:03:50.091713 29304 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:50.088294 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:54815 (local address 127.18.96.254:44919)
0501 02:03:50.088462 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:50.088467 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:50.089362 (+ 895us) server_negotiation.cc:408] Connection header received
0501 02:03:50.090118 (+ 756us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:50.090122 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:50.090163 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:50.090244 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:50.091258 (+ 1014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:50.091261 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:50.091264 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:50.091281 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:50.091300 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:50.091308 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:50.091414 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:50.091554 (+ 140us) server_negotiation.cc:300] Negotiation successful
0501 02:03:50.091587 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260501 02:03:50.092347 29585 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:50.088546 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44919 (local address 127.18.96.194:54815)
0501 02:03:50.089229 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:50.089267 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:50.089899 (+ 632us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:50.090386 (+ 487us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:50.090393 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:50.090443 (+ 50us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:50.091033 (+ 590us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:50.091034 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:50.091053 (+ 19us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:50.091093 (+ 40us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:50.091405 (+ 312us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:50.091411 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:50.091416 (+ 5us) client_negotiation.cc:770] Sending connection context
0501 02:03:50.091589 (+ 173us) client_negotiation.cc:241] Negotiation successful
0501 02:03:50.091676 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260501 02:03:50.093513 29583 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44919
I20260501 02:03:50.093713 29583 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:50.094169 29583 heartbeater.cc:507] Master 127.18.96.254:44919 requested a full tablet report, sending...
I20260501 02:03:50.095183 29264 ts_manager.cc:194] Registered new tserver with Master: 9f464c50d5b84104a9b58f0ff24ebc4d (127.18.96.194:44663)
W20260501 02:03:50.187139 29586 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:50.187352 29586 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:50.187436 29586 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:50.190564 29586 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:50.190639 29586 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:50.190737 29586 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:50.194267 29586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:50.195200 29586 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:50.196189 29586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:50.202211 29594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:50.202277 29591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:50.202319 29586 server_base.cc:1061] running on GCE node
W20260501 02:03:50.202212 29592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:50.202756 29586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:50.203236 29586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:50.204406 29586 hybrid_clock.cc:648] HybridClock initialized: now 1777601030204386 us; error 42 us; skew 500 ppm
I20260501 02:03:50.206067 29586 webserver.cc:492] Webserver started at http://127.18.96.195:32875/ using document root <none> and password file <none>
I20260501 02:03:50.206593 29586 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:50.206672 29586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:50.206883 29586 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:50.208389 29586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "d8f52db739eb44de81e19156268d9982"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "40e5bdfd67ca59c0eaf8366b5ee32480"
server_key_iv: "f1f5e42c6d437e00b704f65083e4e074"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.208838 29586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f52db739eb44de81e19156268d9982"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "40e5bdfd67ca59c0eaf8366b5ee32480"
server_key_iv: "f1f5e42c6d437e00b704f65083e4e074"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.211803 29586 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:03:50.213688 29600 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.214541 29586 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:50.214661 29586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "d8f52db739eb44de81e19156268d9982"
format_stamp: "Formatted at 2026-05-01 02:03:50 on dist-test-slave-2j90"
server_key: "40e5bdfd67ca59c0eaf8366b5ee32480"
server_key_iv: "f1f5e42c6d437e00b704f65083e4e074"
server_key_version: "encryptionkey@0"
I20260501 02:03:50.214752 29586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:50.219044 29586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:50.219551 29586 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:50.219744 29586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:50.220234 29586 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:50.221048 29586 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:50.221124 29586 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.221210 29586 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:50.221258 29586 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:50.230151 29586 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:40565
I20260501 02:03:50.230209 29713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:40565 every 8 connection(s)
I20260501 02:03:50.230942 29586 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:03:50.235737 29304 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:50.232513 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:54187 (local address 127.18.96.254:44919)
0501 02:03:50.232650 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:50.232654 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:50.233344 (+ 690us) server_negotiation.cc:408] Connection header received
0501 02:03:50.234114 (+ 770us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:50.234118 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:50.234180 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:50.234270 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:50.235242 (+ 972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:50.235246 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:50.235249 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:03:50.235261 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:50.235285 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:03:50.235298 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:50.235394 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:50.235562 (+ 168us) server_negotiation.cc:300] Negotiation successful
0501 02:03:50.235599 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:03:50.236111 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29586
I20260501 02:03:50.236239 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:50.236337 29716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:50.232728 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44919 (local address 127.18.96.195:54187)
0501 02:03:50.233209 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:50.233237 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:50.233886 (+ 649us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:50.234402 (+ 516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:50.234419 (+ 17us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:50.234474 (+ 55us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:50.235030 (+ 556us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:03:50.235032 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:50.235054 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:03:50.235097 (+ 43us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:50.235389 (+ 292us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:50.235395 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:50.235402 (+ 7us) client_negotiation.cc:770] Sending connection context
0501 02:03:50.235593 (+ 191us) client_negotiation.cc:241] Negotiation successful
0501 02:03:50.235686 (+ 93us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260501 02:03:50.236521 18819 external_mini_cluster.cc:1468] Setting key 6acf97d74de073eac0d21c4174c90eaa
I20260501 02:03:50.237560 29714 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44919
I20260501 02:03:50.237775 29714 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:50.238214 29714 heartbeater.cc:507] Master 127.18.96.254:44919 requested a full tablet report, sending...
I20260501 02:03:50.239207 29264 ts_manager.cc:194] Registered new tserver with Master: d8f52db739eb44de81e19156268d9982 (127.18.96.195:40565)
I20260501 02:03:50.250245 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260501 02:03:50.253098 29304 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:03:50.252163 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.18.96.254:44919)
0501 02:03:50.252319 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:50.252326 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:50.252513 (+ 187us) server_negotiation.cc:408] Connection header received
0501 02:03:50.252648 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:50.252650 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:50.252683 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:50.252750 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:50.252968 (+ 218us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43734: BlockingRecv error: recv got EOF from 127.0.0.1:43734 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:03:50.253295 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29324
I20260501 02:03:50.259259 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29455
I20260501 02:03:50.264613 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29586
I20260501 02:03:50.269800 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29233
2026-05-01T02:03:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (700 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:50 dist-test-slave-2j90 krb5kdc[29724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:50 dist-test-slave-2j90 krb5kdc[29724](info): set up 2 sockets
May 01 02:03:50 dist-test-slave-2j90 krb5kdc[29724](info): commencing operation
krb5kdc: starting...
W20260501 02:03:52.242758 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:52.373133 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33591
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:33591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:52.469259 29740 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:52.469475 29740 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:52.469553 29740 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:52.472883 29740 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:03:52.472976 29740 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:52.473022 29740 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:52.473084 29740 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:03:52.473136 29740 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:03:52.477278 29740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:33591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:52.478315 29740 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:52.479077 29740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:52.484402 29745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:52.484380 29746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:52.484372 29748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:52.484618 29740 server_base.cc:1061] running on GCE node
I20260501 02:03:52.485183 29740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:52.486034 29740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:52.487254 29740 hybrid_clock.cc:648] HybridClock initialized: now 1777601032487250 us; error 34 us; skew 500 ppm
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:52.490161 29740 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:03:52.491159 29740 webserver.cc:492] Webserver started at http://127.18.96.254:46295/ using document root <none> and password file <none>
I20260501 02:03:52.491737 29740 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:52.491806 29740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:52.491995 29740 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:52.493530 29740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "9484addff9e44f34b3eb8ef4cb719702"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "d2801847452efb3a425e371943bcb5c0"
server_key_iv: "1876e05715b4775b142d6fb3c12afcc4"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.494021 29740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "9484addff9e44f34b3eb8ef4cb719702"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "d2801847452efb3a425e371943bcb5c0"
server_key_iv: "1876e05715b4775b142d6fb3c12afcc4"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.497157 29740 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260501 02:03:52.499194 29755 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:52.500057 29740 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:52.500208 29740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9484addff9e44f34b3eb8ef4cb719702"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "d2801847452efb3a425e371943bcb5c0"
server_key_iv: "1876e05715b4775b142d6fb3c12afcc4"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.500353 29740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:52.509369 29740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:52.516459 29740 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:52.516618 29740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:52.523749 29807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:33591 every 8 connection(s)
I20260501 02:03:52.523747 29740 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:33591
I20260501 02:03:52.524876 29740 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:03:52.527468 29808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:52.528527 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29740
I20260501 02:03:52.528659 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:03:52.528951 18819 external_mini_cluster.cc:1468] Setting key f8aa326d6f04d11068741d3369969fea
I20260501 02:03:52.533452 29808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: Bootstrap starting.
I20260501 02:03:52.535570 29808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:52.536191 29808 log.cc:826] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: Log is configured to *not* fsync() on all Append() calls
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:52.538144 29808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: No bootstrap required, opened a new log
I20260501 02:03:52.541119 29808 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } }
I20260501 02:03:52.541318 29808 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:52.541393 29808 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9484addff9e44f34b3eb8ef4cb719702, State: Initialized, Role: FOLLOWER
I20260501 02:03:52.541842 29811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:52.530270 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.18.96.254:33591)
0501 02:03:52.530690 (+ 420us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:52.530702 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:52.530739 (+ 37us) server_negotiation.cc:408] Connection header received
0501 02:03:52.531386 (+ 647us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:52.531408 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:52.531717 (+ 309us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:52.532032 (+ 315us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:52.532853 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.533994 (+ 1141us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:52.534677 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.534916 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:52.537399 (+ 2483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:52.537417 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:52.537430 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:52.537465 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:52.539439 (+ 1974us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:52.539837 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:52.539843 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:52.539848 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:52.539924 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:52.540164 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:52.540167 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:52.540169 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:52.540514 (+ 345us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:52.540698 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:52.540939 (+ 241us) server_negotiation.cc:300] Negotiation successful
0501 02:03:52.541146 (+ 207us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260501 02:03:52.541787 29808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } }
I20260501 02:03:52.541944 29808 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:03:52.542022 29808 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:03:52.542110 29808 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:52.543347 29808 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } }
I20260501 02:03:52.543706 29808 leader_election.cc:304] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9484addff9e44f34b3eb8ef4cb719702; no voters:
I20260501 02:03:52.544060 29808 leader_election.cc:290] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:03:52.544178 29813 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:52.544390 29813 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [term 1 LEADER]: Becoming Leader. State: Replica: 9484addff9e44f34b3eb8ef4cb719702, State: Running, Role: LEADER
I20260501 02:03:52.544669 29813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } }
I20260501 02:03:52.545084 29808 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:03:52.546203 29815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9484addff9e44f34b3eb8ef4cb719702. Latest consensus state: current_term: 1 leader_uuid: "9484addff9e44f34b3eb8ef4cb719702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } } }
I20260501 02:03:52.546319 29815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:52.546617 29814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9484addff9e44f34b3eb8ef4cb719702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9484addff9e44f34b3eb8ef4cb719702" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33591 } } }
I20260501 02:03:52.546701 29814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702 [sys.catalog]: This master's current role is: LEADER
I20260501 02:03:52.546797 29822 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:03:52.549630 29822 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:03:52.554401 29822 catalog_manager.cc:1357] Generated new cluster ID: 00a817101e1e4a70b63b0d376d6b60cd
I20260501 02:03:52.554481 29822 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:03:52.567210 29822 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:03:52.568090 29822 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:03:52.574841 29822 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: Generated new TSK 0
I20260501 02:03:52.575292 29822 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:52.620600 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33591
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:52.716573 29836 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:52.716765 29836 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:52.716837 29836 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:52.720155 29836 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:52.720207 29836 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:52.720279 29836 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:03:52.724364 29836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:52.725310 29836 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:52.726171 29836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:52.732074 29841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:52.732172 29844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:52.732286 29836 server_base.cc:1061] running on GCE node
W20260501 02:03:52.732329 29842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:52.732661 29836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:52.733183 29836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:52.734349 29836 hybrid_clock.cc:648] HybridClock initialized: now 1777601032734343 us; error 27 us; skew 500 ppm
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:52.736840 29836 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:03:52.737774 29836 webserver.cc:492] Webserver started at http://127.18.96.193:46713/ using document root <none> and password file <none>
I20260501 02:03:52.738338 29836 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:52.738423 29836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:52.738619 29836 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:52.740145 29836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "aa1d6ca8da634ee0b945766063ec5a81"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "20c3c120d4c08194d5dacdbbef6f3503"
server_key_iv: "151f7a276caece035f3d3cd5bd22a44d"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.740595 29836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "aa1d6ca8da634ee0b945766063ec5a81"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "20c3c120d4c08194d5dacdbbef6f3503"
server_key_iv: "151f7a276caece035f3d3cd5bd22a44d"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.743561 29836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:03:52.745813 29851 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:52.746773 29836 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:52.746896 29836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "aa1d6ca8da634ee0b945766063ec5a81"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "20c3c120d4c08194d5dacdbbef6f3503"
server_key_iv: "151f7a276caece035f3d3cd5bd22a44d"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.747002 29836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:52.787355 29836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:52.789834 29836 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:52.790046 29836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:52.790625 29836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:52.791482 29836 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:52.791555 29836 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:52.791625 29836 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:52.791669 29836 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:52.800146 29836 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:33775
I20260501 02:03:52.800177 29964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:33775 every 8 connection(s)
I20260501 02:03:52.801041 29836 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:03:52.805747 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29836
I20260501 02:03:52.805887 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:03:52.806180 18819 external_mini_cluster.cc:1468] Setting key 0ae9eb0afeeaabbefff0e791c5451f29
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:52.813705 29811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:52.802719 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:52555 (local address 127.18.96.254:33591)
0501 02:03:52.802856 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:52.802860 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:52.803463 (+ 603us) server_negotiation.cc:408] Connection header received
0501 02:03:52.804319 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:52.804323 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:52.804392 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:52.804490 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:52.805598 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.806247 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:52.806948 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.807133 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:52.810105 (+ 2972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:52.810126 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:52.810133 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:52.810167 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:52.811792 (+ 1625us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:52.812242 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:52.812249 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:52.812254 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:52.812309 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:52.812621 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:52.812627 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:52.812631 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:52.812839 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:52.812948 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:52.813382 (+ 434us) server_negotiation.cc:300] Negotiation successful
0501 02:03:52.813518 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:03:52.814153 29967 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:52.802957 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33591 (local address 127.18.96.193:52555)
0501 02:03:52.803339 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:52.803370 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:52.804134 (+ 764us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:52.804633 (+ 499us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:52.804641 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:52.804972 (+ 331us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:52.805403 (+ 431us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:52.805413 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.806369 (+ 956us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:52.806375 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:52.806838 (+ 463us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:52.806845 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:52.807042 (+ 197us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:52.808223 (+ 1181us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:52.808249 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:52.809908 (+ 1659us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:52.811906 (+ 1998us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:52.811911 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:52.811918 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:52.812138 (+ 220us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:52.812429 (+ 291us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:52.812431 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:52.812433 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:52.812510 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:52.812952 (+ 442us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:52.812956 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:52.813137 (+ 181us) client_negotiation.cc:770] Sending connection context
0501 02:03:52.813291 (+ 154us) client_negotiation.cc:241] Negotiation successful
0501 02:03:52.813463 (+ 172us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260501 02:03:52.815209 29965 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33591
I20260501 02:03:52.815438 29965 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:03:52.815929 29965 heartbeater.cc:507] Master 127.18.96.254:33591 requested a full tablet report, sending...
I20260501 02:03:52.817291 29772 ts_manager.cc:194] Registered new tserver with Master: aa1d6ca8da634ee0b945766063ec5a81 (127.18.96.193:33775)
I20260501 02:03:52.818452 29772 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:52555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:52.854823 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33591
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:52.954002 29972 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:52.954209 29972 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:52.954313 29972 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:52.957545 29972 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:52.957597 29972 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:52.957706 29972 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:03:52.961777 29972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:52.962745 29972 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:52.963501 29972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:52.969458 29977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:52.969525 29980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:52.969537 29978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:52.969826 29972 server_base.cc:1061] running on GCE node
I20260501 02:03:52.970161 29972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:52.970702 29972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:52.971892 29972 hybrid_clock.cc:648] HybridClock initialized: now 1777601032971875 us; error 40 us; skew 500 ppm
May 01 02:03:52 dist-test-slave-2j90 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:52.974472 29972 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:03:52.975378 29972 webserver.cc:492] Webserver started at http://127.18.96.194:44827/ using document root <none> and password file <none>
I20260501 02:03:52.975869 29972 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:52.975950 29972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:52.976166 29972 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:52.977659 29972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "e5675a877f934cd989feb5869a393ccc"
server_key_iv: "f9da38216f157a4de48f0b68bbfefdf8"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.978161 29972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "e5675a877f934cd989feb5869a393ccc"
server_key_iv: "f9da38216f157a4de48f0b68bbfefdf8"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.981155 29972 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:03:52.983103 29987 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:52.983989 29972 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:03:52.984114 29972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f"
format_stamp: "Formatted at 2026-05-01 02:03:52 on dist-test-slave-2j90"
server_key: "e5675a877f934cd989feb5869a393ccc"
server_key_iv: "f9da38216f157a4de48f0b68bbfefdf8"
server_key_version: "encryptionkey@0"
I20260501 02:03:52.984207 29972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:52.999890 29972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:53.002116 29972 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:53.002290 29972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:53.002781 29972 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:53.003563 29972 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:53.003631 29972 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:53.003715 29972 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:53.003767 29972 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:53.012645 29972 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42271
I20260501 02:03:53.012679 30100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42271 every 8 connection(s)
I20260501 02:03:53.013468 29972 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:03:53.019559 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 29972
I20260501 02:03:53.019687 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:03:53.019948 18819 external_mini_cluster.cc:1468] Setting key cf4d70ad55b966f3a3d49facb01316e6
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[29724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601032, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:53.025478 29811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.015141 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:38981 (local address 127.18.96.254:33591)
0501 02:03:53.015282 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.015286 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.015865 (+ 579us) server_negotiation.cc:408] Connection header received
0501 02:03:53.016669 (+ 804us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.016673 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.016750 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.016856 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:53.017902 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.018579 (+ 677us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.019204 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.019385 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.021894 (+ 2509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:53.021935 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:53.021938 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:53.022003 (+ 65us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:53.023713 (+ 1710us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.024177 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.024182 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.024184 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.024229 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.024517 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.024521 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.024523 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.024668 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:53.024768 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.025196 (+ 428us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.025313 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:03:53.026021 30103 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.015381 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33591 (local address 127.18.96.194:38981)
0501 02:03:53.015744 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.015772 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.016488 (+ 716us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.016996 (+ 508us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.017004 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.017352 (+ 348us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:53.017757 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.017768 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.018694 (+ 926us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.018699 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.019080 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.019086 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.019287 (+ 201us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.020351 (+ 1064us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:53.020369 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:53.021747 (+ 1378us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:53.023812 (+ 2065us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:53.023818 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:53.023834 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:53.024068 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:53.024328 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:53.024332 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:53.024335 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:53.024424 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:53.024779 (+ 355us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:53.024786 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:53.024963 (+ 177us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.025138 (+ 175us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.025334 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260501 02:03:53.026926 30101 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33591
I20260501 02:03:53.027210 30101 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:53.027751 30101 heartbeater.cc:507] Master 127.18.96.254:33591 requested a full tablet report, sending...
I20260501 02:03:53.028798 29772 ts_manager.cc:194] Registered new tserver with Master: e5e0dcc6e4c94e26ad22fc8efac0af1f (127.18.96.194:42271)
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:03:53.029410 29772 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:38981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:03:53.066404 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33591
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:03:53.166774 30108 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:03:53.166985 30108 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:03:53.167176 30108 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:03:53.170477 30108 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:03:53.170526 30108 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:03:53.170646 30108 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:03:53.174602 30108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:03:53.175510 30108 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:03:53.176332 30108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:03:53.182597 30116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:03:53.182683 30114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:53.182847 30108 server_base.cc:1061] running on GCE node
W20260501 02:03:53.182889 30113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:03:53.183241 30108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:03:53.183779 30108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:03:53.184947 30108 hybrid_clock.cc:648] HybridClock initialized: now 1777601033184934 us; error 33 us; skew 500 ppm
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[29724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:03:53.187546 30108 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:03:53.188503 30108 webserver.cc:492] Webserver started at http://127.18.96.195:45209/ using document root <none> and password file <none>
I20260501 02:03:53.189016 30108 fs_manager.cc:362] Metadata directory not provided
I20260501 02:03:53.189085 30108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:03:53.189318 30108 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:03:53.190912 30108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "c6671260821848b9a10e2be5c58eabb8"
format_stamp: "Formatted at 2026-05-01 02:03:53 on dist-test-slave-2j90"
server_key: "1feb78bafd060d06a2796e3f8451bd1c"
server_key_iv: "a17bd618ff85114bb950b1e905b105fc"
server_key_version: "encryptionkey@0"
I20260501 02:03:53.191426 30108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "c6671260821848b9a10e2be5c58eabb8"
format_stamp: "Formatted at 2026-05-01 02:03:53 on dist-test-slave-2j90"
server_key: "1feb78bafd060d06a2796e3f8451bd1c"
server_key_iv: "a17bd618ff85114bb950b1e905b105fc"
server_key_version: "encryptionkey@0"
I20260501 02:03:53.194455 30108 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:03:53.196394 30123 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:53.197286 30108 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:03:53.197413 30108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "c6671260821848b9a10e2be5c58eabb8"
format_stamp: "Formatted at 2026-05-01 02:03:53 on dist-test-slave-2j90"
server_key: "1feb78bafd060d06a2796e3f8451bd1c"
server_key_iv: "a17bd618ff85114bb950b1e905b105fc"
server_key_version: "encryptionkey@0"
I20260501 02:03:53.197510 30108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:03:53.208228 30108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:03:53.210490 30108 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:03:53.210669 30108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:03:53.211151 30108 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:03:53.211921 30108 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:03:53.211992 30108 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:53.212051 30108 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:03:53.212101 30108 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:03:53.220535 30108 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:40855
I20260501 02:03:53.220546 30236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:40855 every 8 connection(s)
I20260501 02:03:53.221346 30108 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[29724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601033, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:03:53.231561 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 30108
I20260501 02:03:53.231705 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:03:53.232021 18819 external_mini_cluster.cc:1468] Setting key 35c15290d72c272c88534415ae7b9736
I20260501 02:03:53.232147 29811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.222909 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:38033 (local address 127.18.96.254:33591)
0501 02:03:53.223070 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.223074 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.223623 (+ 549us) server_negotiation.cc:408] Connection header received
0501 02:03:53.224356 (+ 733us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.224359 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.224408 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.224482 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:53.225479 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.226017 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.226588 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.226721 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.229125 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:53.229143 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:53.229145 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:53.229167 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:53.230436 (+ 1269us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.230818 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.230820 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.230821 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.230855 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.231060 (+ 205us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.231061 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.231062 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.231203 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:53.231269 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.231871 (+ 602us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.231982 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260501 02:03:53.232841 30239 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.223136 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33591 (local address 127.18.96.195:38033)
0501 02:03:53.223496 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.223525 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.224197 (+ 672us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.224618 (+ 421us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.224624 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.224939 (+ 315us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:03:53.225358 (+ 419us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.225368 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.226128 (+ 760us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.226131 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.226498 (+ 367us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.226503 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.226648 (+ 145us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.227587 (+ 939us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:03:53.227603 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:03:53.228984 (+ 1381us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:03:53.230532 (+ 1548us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:53.230536 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:53.230544 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:53.230739 (+ 195us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:53.230924 (+ 185us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:03:53.230926 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:03:53.230927 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:03:53.230994 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:03:53.231280 (+ 286us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:03:53.231284 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:03:53.231539 (+ 255us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.231802 (+ 263us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.232027 (+ 225us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260501 02:03:53.233758 30237 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33591
I20260501 02:03:53.234025 30237 heartbeater.cc:461] Registering TS with master...
I20260501 02:03:53.234452 30237 heartbeater.cc:507] Master 127.18.96.254:33591 requested a full tablet report, sending...
I20260501 02:03:53.235373 29771 ts_manager.cc:194] Registered new tserver with Master: c6671260821848b9a10e2be5c58eabb8 (127.18.96.195:40855)
I20260501 02:03:53.235858 29771 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:38033
I20260501 02:03:53.245709 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:03:53.254161 29811 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.247697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.18.96.254:33591)
0501 02:03:53.247858 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.247862 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.247991 (+ 129us) server_negotiation.cc:408] Connection header received
0501 02:03:53.248151 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.248154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.248202 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.248284 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:03:53.249082 (+ 798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.249561 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.250242 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.250443 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.251482 (+ 1039us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:03:53.251498 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:03:53.251501 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:03:53.251533 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:03:53.252672 (+ 1139us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.253045 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.253048 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.253052 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.253098 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:03:53.253340 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:03:53.253343 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:03:53.253345 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:03:53.253498 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:03:53.253634 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.253869 (+ 235us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.253974 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260501 02:03:53.256558 29771 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:03:53.258986 29771 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:03:53.271557 30248 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.266910 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42271 (local address 127.0.0.1:57240)
0501 02:03:53.267628 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.267662 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.267811 (+ 149us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.268559 (+ 748us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.268564 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.268584 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:53.268864 (+ 280us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.268875 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270222 (+ 1347us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.270225 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.271116 (+ 891us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.271123 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.271220 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.271261 (+ 41us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.271344 (+ 83us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.271420 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260501 02:03:53.272270 30252 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.267622 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.18.96.195:40855)
0501 02:03:53.268171 (+ 549us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.268177 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.268514 (+ 337us) server_negotiation.cc:408] Connection header received
0501 02:03:53.268651 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.268655 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.268802 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.268943 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:53.269306 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270363 (+ 1057us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.271424 (+ 1061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.271920 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.272016 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.272073 (+ 57us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.272142 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":135,"threads_started":1}
I20260501 02:03:53.272325 30251 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.267614 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:33775 (local address 127.0.0.1:58782)
0501 02:03:53.268461 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.268473 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.268625 (+ 152us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.269225 (+ 600us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.269231 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.269253 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:53.269434 (+ 181us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.269438 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270521 (+ 1083us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.270525 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.271296 (+ 771us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.271304 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.272074 (+ 770us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.272083 (+ 9us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.272173 (+ 90us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.272218 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260501 02:03:53.272433 30253 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.267931 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.18.96.193:33775)
0501 02:03:53.268309 (+ 378us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.268315 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.268542 (+ 227us) server_negotiation.cc:408] Connection header received
0501 02:03:53.268779 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.268785 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.268950 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.269060 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:53.269542 (+ 482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270399 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.271452 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.271914 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.272017 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.272213 (+ 196us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.272287 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":108,"threads_started":1}
I20260501 02:03:53.272482 30249 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.267075 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57240 (local address 127.18.96.194:42271)
0501 02:03:53.267648 (+ 573us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.267656 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.267934 (+ 278us) server_negotiation.cc:408] Connection header received
0501 02:03:53.268030 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.268037 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.268249 (+ 212us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.268392 (+ 143us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:53.268991 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270101 (+ 1110us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.271644 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.272126 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.272200 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.272252 (+ 52us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.272325 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":95,"threads_started":1}
I20260501 02:03:53.272691 30250 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.267155 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:40855 (local address 127.0.0.1:34964)
0501 02:03:53.268321 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.268335 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.268427 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.268938 (+ 511us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.268941 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.268957 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:53.269188 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.269196 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.270473 (+ 1277us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.270477 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.271272 (+ 795us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.271284 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.271419 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.271434 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.271483 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.271534 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1087,"thread_start_us":121,"threads_started":1}
I20260501 02:03:53.275236 30171 tablet_service.cc:1511] Processing CreateTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef (DEFAULT_TABLE table=test-table [id=0c60faaef1ec4e31b32dcd5805e7d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:53.275269 30035 tablet_service.cc:1511] Processing CreateTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef (DEFAULT_TABLE table=test-table [id=0c60faaef1ec4e31b32dcd5805e7d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:53.275269 29899 tablet_service.cc:1511] Processing CreateTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef (DEFAULT_TABLE table=test-table [id=0c60faaef1ec4e31b32dcd5805e7d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:03:53.276077 30171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e60302ec6384d1cb500f5438ed1b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:53.276103 30035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e60302ec6384d1cb500f5438ed1b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:53.276104 29899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e60302ec6384d1cb500f5438ed1b9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:03:53.280928 30254 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Bootstrap starting.
I20260501 02:03:53.282135 30255 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Bootstrap starting.
I20260501 02:03:53.283131 30254 tablet_bootstrap.cc:654] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:53.283826 30255 tablet_bootstrap.cc:654] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:53.283819 30254 log.cc:826] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:53.284498 30256 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Bootstrap starting.
I20260501 02:03:53.284621 30255 log.cc:826] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:53.286175 30256 tablet_bootstrap.cc:654] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Neither blocks nor log segments found. Creating new log.
I20260501 02:03:53.286226 30255 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: No bootstrap required, opened a new log
I20260501 02:03:53.286403 30255 ts_tablet_manager.cc:1403] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260501 02:03:53.286842 30256 log.cc:826] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Log is configured to *not* fsync() on all Append() calls
I20260501 02:03:53.288261 30254 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: No bootstrap required, opened a new log
I20260501 02:03:53.288489 30254 ts_tablet_manager.cc:1403] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260501 02:03:53.288594 30256 tablet_bootstrap.cc:492] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: No bootstrap required, opened a new log
I20260501 02:03:53.288802 30256 ts_tablet_manager.cc:1403] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:03:53.289808 30255 raft_consensus.cc:359] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.290030 30255 raft_consensus.cc:385] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:53.290122 30255 raft_consensus.cc:740] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e0dcc6e4c94e26ad22fc8efac0af1f, State: Initialized, Role: FOLLOWER
I20260501 02:03:53.290542 30255 consensus_queue.cc:260] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.291232 30101 heartbeater.cc:499] Master 127.18.96.254:33591 was elected leader, sending a full tablet report...
I20260501 02:03:53.291644 30254 raft_consensus.cc:359] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.291810 30254 raft_consensus.cc:385] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:53.291868 30254 raft_consensus.cc:740] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6671260821848b9a10e2be5c58eabb8, State: Initialized, Role: FOLLOWER
I20260501 02:03:53.292294 30254 consensus_queue.cc:260] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.292272 30256 raft_consensus.cc:359] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.292441 30256 raft_consensus.cc:385] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:03:53.292544 30256 raft_consensus.cc:740] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1d6ca8da634ee0b945766063ec5a81, State: Initialized, Role: FOLLOWER
I20260501 02:03:53.292660 30255 ts_tablet_manager.cc:1434] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260501 02:03:53.292963 30256 consensus_queue.cc:260] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.293138 30254 ts_tablet_manager.cc:1434] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:03:53.293462 30237 heartbeater.cc:499] Master 127.18.96.254:33591 was elected leader, sending a full tablet report...
I20260501 02:03:53.293565 29965 heartbeater.cc:499] Master 127.18.96.254:33591 was elected leader, sending a full tablet report...
I20260501 02:03:53.293736 30256 ts_tablet_manager.cc:1434] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260501 02:03:53.302018 29966 tablet.cc:2404] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:03:53.392859 30260 raft_consensus.cc:493] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:03:53.393066 30260 raft_consensus.cc:515] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.394240 30260 leader_election.cc:290] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6671260821848b9a10e2be5c58eabb8 (127.18.96.195:40855), aa1d6ca8da634ee0b945766063ec5a81 (127.18.96.193:33775)
I20260501 02:03:53.397410 30103 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.394481 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:33775 (local address 127.18.96.194:51369)
0501 02:03:53.394598 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.394614 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.394704 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.394997 (+ 293us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.395001 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.395027 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:53.395320 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.395325 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.396239 (+ 914us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.396242 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.397020 (+ 778us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.397028 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.397154 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.397171 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.397220 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.397271 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260501 02:03:53.397791 30253 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.394584 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:51369 (local address 127.18.96.193:33775)
0501 02:03:53.394720 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.394726 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.394744 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:03:53.394800 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.394803 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.394860 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.394946 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:53.395471 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.396126 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.397153 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.397522 (+ 369us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.397576 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.397626 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.397675 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:03:53.397794 30263 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.394481 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:40855 (local address 127.18.96.194:50545)
0501 02:03:53.395079 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.395093 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:03:53.395204 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:03:53.395493 (+ 289us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:03:53.395495 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:03:53.395513 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:03:53.395721 (+ 208us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.395725 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.396559 (+ 834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.396563 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:03:53.397400 (+ 837us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:03:53.397409 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.397520 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.397546 (+ 26us) client_negotiation.cc:770] Sending connection context
0501 02:03:53.397604 (+ 58us) client_negotiation.cc:241] Negotiation successful
0501 02:03:53.397663 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":291,"thread_start_us":126,"threads_started":1}
I20260501 02:03:53.398437 30252 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.394591 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:50545 (local address 127.18.96.195:40855)
0501 02:03:53.394715 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.394718 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.395201 (+ 483us) server_negotiation.cc:408] Connection header received
0501 02:03:53.395310 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.395312 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.395355 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.395470 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:03:53.395836 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.396462 (+ 626us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.397717 (+ 1255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.398235 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.398264 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.398305 (+ 41us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.398344 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:03:53.398427 29919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e60302ec6384d1cb500f5438ed1b9ef" candidate_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1d6ca8da634ee0b945766063ec5a81" is_pre_election: true
I20260501 02:03:53.398705 29919 raft_consensus.cc:2468] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e0dcc6e4c94e26ad22fc8efac0af1f in term 0.
I20260501 02:03:53.399003 30191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e60302ec6384d1cb500f5438ed1b9ef" candidate_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6671260821848b9a10e2be5c58eabb8" is_pre_election: true
I20260501 02:03:53.399194 29990 leader_election.cc:304] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1d6ca8da634ee0b945766063ec5a81, e5e0dcc6e4c94e26ad22fc8efac0af1f; no voters:
I20260501 02:03:53.399258 30191 raft_consensus.cc:2468] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e0dcc6e4c94e26ad22fc8efac0af1f in term 0.
I20260501 02:03:53.399451 30260 raft_consensus.cc:2804] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:03:53.399539 30260 raft_consensus.cc:493] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:03:53.399616 30260 raft_consensus.cc:3060] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:53.400542 30260 raft_consensus.cc:515] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.400897 30260 leader_election.cc:290] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [CANDIDATE]: Term 1 election: Requested vote from peers c6671260821848b9a10e2be5c58eabb8 (127.18.96.195:40855), aa1d6ca8da634ee0b945766063ec5a81 (127.18.96.193:33775)
I20260501 02:03:53.401271 29919 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e60302ec6384d1cb500f5438ed1b9ef" candidate_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1d6ca8da634ee0b945766063ec5a81"
I20260501 02:03:53.401266 30191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e60302ec6384d1cb500f5438ed1b9ef" candidate_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6671260821848b9a10e2be5c58eabb8"
I20260501 02:03:53.401410 29919 raft_consensus.cc:3060] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:53.401463 30191 raft_consensus.cc:3060] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:03:53.402447 29919 raft_consensus.cc:2468] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e0dcc6e4c94e26ad22fc8efac0af1f in term 1.
I20260501 02:03:53.402485 30191 raft_consensus.cc:2468] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e0dcc6e4c94e26ad22fc8efac0af1f in term 1.
I20260501 02:03:53.402769 29990 leader_election.cc:304] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1d6ca8da634ee0b945766063ec5a81, e5e0dcc6e4c94e26ad22fc8efac0af1f; no voters:
I20260501 02:03:53.402944 30260 raft_consensus.cc:2804] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:03:53.403191 30260 raft_consensus.cc:697] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 1 LEADER]: Becoming Leader. State: Replica: e5e0dcc6e4c94e26ad22fc8efac0af1f, State: Running, Role: LEADER
I20260501 02:03:53.403482 30260 consensus_queue.cc:237] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } }
I20260501 02:03:53.406493 29770 catalog_manager.cc:5671] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e0dcc6e4c94e26ad22fc8efac0af1f (127.18.96.194). New cstate: current_term: 1 leader_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e0dcc6e4c94e26ad22fc8efac0af1f" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 } health_report { overall_health: UNKNOWN } } }
I20260501 02:03:53.420847 30249 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.418436 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.18.96.194:42271)
0501 02:03:53.418563 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:03:53.418566 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:03:53.418577 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:03:53.418674 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:03:53.418676 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:03:53.418721 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:03:53.418796 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:03:53.419079 (+ 283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.419529 (+ 450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:03:53.420094 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:03:53.420221 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:03:53.420284 (+ 63us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:03:53.420577 (+ 293us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:03:53.420653 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:03:53.420730 (+ 77us) server_negotiation.cc:300] Negotiation successful
0501 02:03:53.420764 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260501 02:03:53.425470 30191 raft_consensus.cc:1275] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 1 FOLLOWER]: Refusing update from remote peer e5e0dcc6e4c94e26ad22fc8efac0af1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:53.425472 29919 raft_consensus.cc:1275] T 1e60302ec6384d1cb500f5438ed1b9ef P aa1d6ca8da634ee0b945766063ec5a81 [term 1 FOLLOWER]: Refusing update from remote peer e5e0dcc6e4c94e26ad22fc8efac0af1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:03:53.426142 30264 consensus_queue.cc:1048] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1d6ca8da634ee0b945766063ec5a81" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:53.426317 30260 consensus_queue.cc:1048] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6671260821848b9a10e2be5c58eabb8" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 40855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:03:53.433015 30269 mvcc.cc:204] Tried to move back new op lower bound from 7281053832904847360 to 7281053832823406592. Current Snapshot: MvccSnapshot[applied={T|T < 7281053832904847360}]
I20260501 02:03:53.436133 30268 mvcc.cc:204] Tried to move back new op lower bound from 7281053832904847360 to 7281053832823406592. Current Snapshot: MvccSnapshot[applied={T|T < 7281053832904847360}]
I20260501 02:03:53.439677 29771 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:53.439837 29771 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:03:53.441897 29771 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9484addff9e44f34b3eb8ef4cb719702: Sending DeleteTablet for 3 replicas of tablet 1e60302ec6384d1cb500f5438ed1b9ef
I20260501 02:03:53.442616 30171 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:34964
I20260501 02:03:53.442708 29899 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:58782
I20260501 02:03:53.443228 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29836
I20260501 02:03:53.443245 30035 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e60302ec6384d1cb500f5438ed1b9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:03:53 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:57240
I20260501 02:03:53.444203 30279 tablet_replica.cc:333] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: stopping tablet replica
I20260501 02:03:53.444312 30277 tablet_replica.cc:333] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: stopping tablet replica
I20260501 02:03:53.444532 30279 raft_consensus.cc:2243] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:03:53.444563 30277 raft_consensus.cc:2243] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:03:53.444885 30277 raft_consensus.cc:2272] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:53.444921 30279 raft_consensus.cc:2272] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:03:53.446221 30279 ts_tablet_manager.cc:1916] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:53.446228 30277 ts_tablet_manager.cc:1916] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:03:53.448705 30277 ts_tablet_manager.cc:1929] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:53.448825 30277 log.cc:1199] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/1e60302ec6384d1cb500f5438ed1b9ef
I20260501 02:03:53.449191 30277 ts_tablet_manager.cc:1950] T 1e60302ec6384d1cb500f5438ed1b9ef P c6671260821848b9a10e2be5c58eabb8: Deleting consensus metadata
I20260501 02:03:53.449460 30279 ts_tablet_manager.cc:1929] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:03:53.449600 30279 log.cc:1199] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/1e60302ec6384d1cb500f5438ed1b9ef
I20260501 02:03:53.449998 30279 ts_tablet_manager.cc:1950] T 1e60302ec6384d1cb500f5438ed1b9ef P e5e0dcc6e4c94e26ad22fc8efac0af1f: Deleting consensus metadata
I20260501 02:03:53.450320 29757 catalog_manager.cc:5002] TS c6671260821848b9a10e2be5c58eabb8 (127.18.96.195:40855): tablet 1e60302ec6384d1cb500f5438ed1b9ef (table test-table [id=0c60faaef1ec4e31b32dcd5805e7d645]) successfully deleted
I20260501 02:03:53.451013 29759 catalog_manager.cc:5002] TS e5e0dcc6e4c94e26ad22fc8efac0af1f (127.18.96.194:42271): tablet 1e60302ec6384d1cb500f5438ed1b9ef (table test-table [id=0c60faaef1ec4e31b32dcd5805e7d645]) successfully deleted
W20260501 02:03:53.452207 29758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:33775 (error 108)
I20260501 02:03:53.452893 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29972
I20260501 02:03:53.453176 30250 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:03:53.452898 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:33775 (local address 127.0.0.1:58794)
0501 02:03:53.453027 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 02:03:53.453083 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:33775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260501 02:03:53.453558 29758 catalog_manager.cc:4729] TS aa1d6ca8da634ee0b945766063ec5a81 (127.18.96.193:33775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e60302ec6384d1cb500f5438ed1b9ef: Network error: Client connection negotiation failed: client connection to 127.18.96.193:33775: connect: Connection refused (error 111)
I20260501 02:03:53.458244 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 30108
I20260501 02:03:53.464097 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 29740
2026-05-01T02:03:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3195 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[30283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[30283](info): set up 2 sockets
May 01 02:03:53 dist-test-slave-2j90 krb5kdc[30283](info): commencing operation
krb5kdc: starting...
W20260501 02:03:55.443190 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.957s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:03:55 dist-test-slave-2j90 krb5kdc[30283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:03:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:03:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangeradmin_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangeradmin_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangerlookup_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangerlookup_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
I20260501 02:03:55.593212 18819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/postgres -l logfile start
2026-05-01 02:03:58.094 UTC [30314] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-05-01 02:03:58.094 UTC [30314] LOG: listening on IPv4 address "127.18.96.212", port 35595
2026-05-01 02:03:58.098 UTC [30314] LOG: listening on Unix socket "/tmp/.s.PGSQL.35595"
2026-05-01 02:03:58.103 UTC [30319] LOG: database system was shut down at 2026-05-01 02:03:56 UTC
2026-05-01 02:03:58.107 UTC [30314] LOG: database system is ready to accept connections
I20260501 02:03:59.994567 18819 mini_postgres.cc:96] Postgres bound to 35595
2026-05-01 02:04:00.000 UTC [30325] FATAL: database "slave" does not exist
127.18.96.212:35595 - accepting connections
I20260501 02:04:00.001091 18819 mini_ranger.cc:161] Starting Ranger...
I20260501 02:04:00.018496 18819 mini_ranger.cc:84] Created miniranger Postgres user
I20260501 02:04:00.085270 18819 mini_ranger.cc:87] Created ranger Postgres database
I20260501 02:04:00.085407 18819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin
2026-05-01 02:04:00,367 [I] DB FLAVOR :POSTGRES
2026-05-01 02:04:00,368 [I] --------- Verifying Ranger DB connection ---------
2026-05-01 02:04:00,368 [I] Checking connection..
2026-05-01 02:04:00,368 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-05-01 02:04:00,683 [I] Checking connection passed.
2026-05-01 02:04:00,684 [I] --------- Verifying version history table ---------
2026-05-01 02:04:00,684 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:04:00,968 [I] Table x_db_version_h does not exist in database ranger
2026-05-01 02:04:00,969 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:04:01,324 [I] Table x_db_version_h does not exist in database ranger
2026-05-01 02:04:01,325 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-05-01 02:04:01,325 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-05-01 02:04:01.636 UTC [30438] WARNING: there is no transaction in progress
2026-05-01 02:04:01,649 [I] create_dbversion_catalog.sql file imported successfully
2026-05-01 02:04:01,649 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:04:01,979 [I] Table x_db_version_h already exists in database 'ranger'
2026-05-01 02:04:01,979 [I] --------- Importing Ranger Core DB Schema ---------
2026-05-01 02:04:01,979 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-05-01 02:04:02,310 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-05-01 02:04:02,609 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2j90.c.gcp-upstream.internal','N') ;"
2026-05-01 02:04:02,922 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-05-01 02:04:03,258 [I] Table x_portal_user does not exist in database ranger
2026-05-01 02:04:03,258 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-05-01 02:04:03,598 [I] Table x_policy_ref_group does not exist in database ranger
2026-05-01 02:04:03,599 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-05-01 02:04:03,906 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-05-01 02:04:04,227 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-05-01 02:04:04,227 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-05-01 02:04:05.021 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.031 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.043 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.053 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.064 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.075 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.120 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.124 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.136 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.145 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.154 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.163 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.170 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.174 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.177 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05.181 UTC [30636] WARNING: there is no transaction in progress
2026-05-01 02:04:05,587 [I] ranger_core_db_postgres.sql file imported successfully
2026-05-01 02:04:05,587 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-05-01 02:04:05,895 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-05-01 02:04:05,895 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-05-01 02:04:06,185 [I] Table x_portal_user already exists in database 'ranger'
2026-05-01 02:04:06,185 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-05-01 02:04:06,500 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-05-01 02:04:06,501 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-05-01 02:04:06,769 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-05-01 02:04:07,060 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2j90.c.gcp-upstream.internal';"
2026-05-01 02:04:07,396 [I] CORE_DB_SCHEMA import status has been updated
2026-05-01 02:04:07,396 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-05-01 02:04:07,700 [I] DB_PATCHES have already been applied
I20260501 02:04:07.705298 18819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskQ5nb4m/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260501 02:04:07.705379 18819 mini_ranger.cc:193] Using host: 127.18.96.212
I20260501 02:04:07.708300 18819 mini_ranger.cc:239] Ranger admin URL: http://127.18.96.212:60217
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2026 2:04:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260501 02:04:09.978096 18819 mini_ranger.cc:160] Time spent starting Ranger: real 9.977s user 0.000s sys 0.009s
May 01, 2026 2:04:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2026 2:04:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
May 01, 2026 2:04:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2026 2:04:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2026 2:04:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260501 02:04:33.636404 18819 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangerkms_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/rangerkms_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 01 02:04:33 dist-test-slave-2j90 krb5kdc[30283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601073, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260501 02:04:33.711373 18819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260501 02:04:33.721122 18819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260501 02:04:33.836858 18819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260501 02:04:33.837025 18819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms
I20260501 02:04:33.837121 18819 mini_ranger_kms.cc:227] Using postgres at 127.18.96.212:35595
2026-05-01 02:04:34,081 [I] DB FLAVOR :POSTGRES
2026-05-01 02:04:34,082 [I] --------- Verifying Ranger DB connection ---------
2026-05-01 02:04:34,082 [I] Checking connection
2026-05-01 02:04:34,082 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-05-01 02:04:34,532 [I] connection success
2026-05-01 02:04:34,532 [I] --------- Verifying Ranger DB tables ---------
2026-05-01 02:04:34,532 [I] Verifying table ranger_masterkey in database rangerkms
2026-05-01 02:04:34,532 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-05-01 02:04:34,881 [I] Table ranger_masterkey does not exist in database rangerkms
2026-05-01 02:04:34,882 [I] --------- Importing Ranger Core DB Schema ---------
2026-05-01 02:04:34,882 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-05-01 02:04:34,882 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:35595/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-05-01 02:04:35,193 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260501 02:04:35.753301 18819 mini_ranger_kms.cc:326] Created kms service
I20260501 02:04:35.956368 18819 mini_ranger_kms.cc:342] Created kudu user
I20260501 02:04:36.028173 18819 mini_ranger_kms.cc:359] Created rangerkms user
May 01 02:04:36 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601073, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:04:36.526700 18819 mini_ranger_kms.cc:400] Added ranger policy
I20260501 02:04:36.526887 18819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260501 02:04:36.526921 18819 mini_ranger_kms.cc:240] Using host: 127.18.96.212
I20260501 02:04:36.531068 18819 mini_ranger_kms.cc:292] Ranger KMS PID: 30935
I20260501 02:04:36.531167 18819 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.18.96.212:55375
May 01, 2026 2:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2026 2:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2026 2:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 01, 2026 2:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp] .....
May 01, 2026 2:04:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp].
May 01, 2026 2:04:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
May 01 02:04:39 dist-test-slave-2j90 krb5kdc[30283](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 01 02:04:40 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:04:40 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:05:11 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:05:11 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:05:41 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:05:41 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:06:11 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:06:11 dist-test-slave-2j90 krb5kdc[30283](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601079, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:06:36.560424 18819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 122.849s user 0.045s sys 0.258s
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/security-itest.cc:1052: Failure
Failed
Bad status: Timed out: Failed to start the Ranger KMS service: process with PID 30935 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p30935
f173
n127.0.0.1:35080->127.18.96.212:35595'
I20260501 02:06:36.561911 18819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260501 02:06:36.582522 18819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260501 02:06:36.582681 18819 mini_ranger.cc:66] Stopping Ranger...
I20260501 02:06:36.928157 18819 mini_ranger.cc:68] Stopped Ranger
2026-05-01 02:06:36.928 UTC [30314] LOG: received smart shutdown request
2026-05-01 02:06:36.931 UTC [30314] LOG: background worker "logical replication launcher" (PID 30322) exited with exit code 1
2026-05-01 02:06:36.931 UTC [30317] LOG: shutting down
2026-05-01 02:06:36.933 UTC [30317] LOG: checkpoint starting: shutdown immediate
2026-05-01 02:06:37.704 UTC [30317] LOG: checkpoint complete: wrote 2479 buffers (15.1%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.037 s, sync=0.727 s, total=0.773 s; sync files=1124, longest=0.006 s, average=0.001 s; distance=12593 kB, estimate=12593 kB; lsn=0/20C6E70, redo lsn=0/20C6E70
2026-05-01 02:06:37.713 UTC [30314] LOG: database system is shut down
2026-05-01T02:06:37Z chronyd exiting
I20260501 02:06:37.728952 18819 test_util.cc:182] -----------------------------------------------
I20260501 02:06:37.729020 18819 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777600936179445-18819-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (164244 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:06:37 dist-test-slave-2j90 krb5kdc[31112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:06:37 dist-test-slave-2j90 krb5kdc[31112](info): set up 2 sockets
May 01 02:06:37 dist-test-slave-2j90 krb5kdc[31112](info): commencing operation
krb5kdc: starting...
W20260501 02:06:39.706017 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.975s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:06:39 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:06:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:06:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangeradmin_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangeradmin_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangerlookup_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangerlookup_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
I20260501 02:06:39.867024 18819 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/postgres -l logfile start
2026-05-01 02:06:42.472 UTC [31143] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-05-01 02:06:42.472 UTC [31143] LOG: listening on IPv4 address "127.18.96.212", port 34579
2026-05-01 02:06:42.476 UTC [31143] LOG: listening on Unix socket "/tmp/.s.PGSQL.34579"
2026-05-01 02:06:42.481 UTC [31148] LOG: database system was shut down at 2026-05-01 02:06:41 UTC
2026-05-01 02:06:42.485 UTC [31143] LOG: database system is ready to accept connections
I20260501 02:06:44.389830 18819 mini_postgres.cc:96] Postgres bound to 34579
2026-05-01 02:06:44.395 UTC [31154] FATAL: database "slave" does not exist
127.18.96.212:34579 - accepting connections
I20260501 02:06:44.396433 18819 mini_ranger.cc:161] Starting Ranger...
I20260501 02:06:44.414187 18819 mini_ranger.cc:84] Created miniranger Postgres user
I20260501 02:06:44.483314 18819 mini_ranger.cc:87] Created ranger Postgres database
I20260501 02:06:44.483390 18819 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin
2026-05-01 02:06:44,774 [I] DB FLAVOR :POSTGRES
2026-05-01 02:06:44,775 [I] --------- Verifying Ranger DB connection ---------
2026-05-01 02:06:44,775 [I] Checking connection..
2026-05-01 02:06:44,775 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-05-01 02:06:45,087 [I] Checking connection passed.
2026-05-01 02:06:45,087 [I] --------- Verifying version history table ---------
2026-05-01 02:06:45,087 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:06:45,437 [I] Table x_db_version_h does not exist in database ranger
2026-05-01 02:06:45,437 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:06:45,735 [I] Table x_db_version_h does not exist in database ranger
2026-05-01 02:06:45,735 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-05-01 02:06:45,735 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-05-01 02:06:46.013 UTC [31267] WARNING: there is no transaction in progress
2026-05-01 02:06:46,029 [I] create_dbversion_catalog.sql file imported successfully
2026-05-01 02:06:46,030 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-05-01 02:06:46,378 [I] Table x_db_version_h already exists in database 'ranger'
2026-05-01 02:06:46,379 [I] --------- Importing Ranger Core DB Schema ---------
2026-05-01 02:06:46,379 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-05-01 02:06:46,717 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-05-01 02:06:46,998 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2j90.c.gcp-upstream.internal','N') ;"
2026-05-01 02:06:47,366 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-05-01 02:06:47,645 [I] Table x_portal_user does not exist in database ranger
2026-05-01 02:06:47,645 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-05-01 02:06:47,954 [I] Table x_policy_ref_group does not exist in database ranger
2026-05-01 02:06:47,955 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-05-01 02:06:48,247 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-05-01 02:06:48,603 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-05-01 02:06:48,603 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-05-01 02:06:49.454 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.465 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.477 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.487 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.497 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.511 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.561 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.566 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.575 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.584 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.593 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.602 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.609 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.613 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.617 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:49.621 UTC [31466] WARNING: there is no transaction in progress
2026-05-01 02:06:50,025 [I] ranger_core_db_postgres.sql file imported successfully
2026-05-01 02:06:50,025 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-05-01 02:06:50,363 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-05-01 02:06:50,363 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-05-01 02:06:50,663 [I] Table x_portal_user already exists in database 'ranger'
2026-05-01 02:06:50,663 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-05-01 02:06:50,982 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-05-01 02:06:50,982 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-05-01 02:06:51,328 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-05-01 02:06:51,621 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2j90.c.gcp-upstream.internal';"
2026-05-01 02:06:51,919 [I] CORE_DB_SCHEMA import status has been updated
2026-05-01 02:06:51,920 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-05-01 02:06:52,215 [I] DB_PATCHES have already been applied
I20260501 02:06:52.221131 18819 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-taskQ5nb4m/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260501 02:06:52.221204 18819 mini_ranger.cc:193] Using host: 127.18.96.212
I20260501 02:06:52.224412 18819 mini_ranger.cc:239] Ranger admin URL: http://127.18.96.212:37141
May 01, 2026 2:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2026 2:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2026 2:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp
May 01, 2026 2:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
May 01, 2026 2:06:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp] .....
May 01, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-admin/ews/webapp].
May 01, 2026 2:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260501 02:06:54.519431 18819 mini_ranger.cc:160] Time spent starting Ranger: real 10.123s user 0.005s sys 0.005s
May 01, 2026 2:07:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
May 01, 2026 2:07:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
May 01, 2026 2:07:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
May 01, 2026 2:07:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
May 01, 2026 2:07:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260501 02:07:18.512850 18819 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangerkms_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/rangerkms_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/HTTP_127.18.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
May 01 02:07:18 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601238, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260501 02:07:18.615865 18819 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260501 02:07:18.627359 18819 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260501 02:07:18.752537 18819 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260501 02:07:18.752640 18819 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms
I20260501 02:07:18.752663 18819 mini_ranger_kms.cc:227] Using postgres at 127.18.96.212:34579
2026-05-01 02:07:18,996 [I] DB FLAVOR :POSTGRES
2026-05-01 02:07:18,997 [I] --------- Verifying Ranger DB connection ---------
2026-05-01 02:07:18,997 [I] Checking connection
2026-05-01 02:07:18,997 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-05-01 02:07:19,416 [I] connection success
2026-05-01 02:07:19,416 [I] --------- Verifying Ranger DB tables ---------
2026-05-01 02:07:19,416 [I] Verifying table ranger_masterkey in database rangerkms
2026-05-01 02:07:19,416 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-05-01 02:07:19,820 [I] Table ranger_masterkey does not exist in database rangerkms
2026-05-01 02:07:19,820 [I] --------- Importing Ranger Core DB Schema ---------
2026-05-01 02:07:19,820 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-05-01 02:07:19,821 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.96.212:34579/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-05-01 02:07:20,198 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260501 02:07:20.714906 18819 mini_ranger_kms.cc:326] Created kms service
I20260501 02:07:20.928249 18819 mini_ranger_kms.cc:342] Created kudu user
I20260501 02:07:20.976970 18819 mini_ranger_kms.cc:359] Created rangerkms user
May 01 02:07:20 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601238, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:21.437958 18819 mini_ranger_kms.cc:400] Added ranger policy
I20260501 02:07:21.438127 18819 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQ5nb4m/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQ5nb4m/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260501 02:07:21.438148 18819 mini_ranger_kms.cc:240] Using host: 127.18.96.212
I20260501 02:07:21.441793 18819 mini_ranger_kms.cc:292] Ranger KMS PID: 31768
I20260501 02:07:21.441871 18819 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.18.96.212:50245
May 01, 2026 2:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
May 01, 2026 2:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
May 01, 2026 2:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
May 01, 2026 2:07:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp] .....
May 01, 2026 2:07:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-kms/ews/webapp].
May 01, 2026 2:07:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260501 02:07:23.714085 18819 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.098s user 0.006s sys 0.204s
I20260501 02:07:23.714215 18819 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260501 02:07:23.714303 18819 mini_ranger_kms.cc:417] 127.18.96.212:50245/kms/v1/keys
May 01 02:07:24 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
May 01 02:07:24 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:07:25 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.96.212@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
May 01 02:07:27 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:27.361042 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33297
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:33297
--ranger_config_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:27.475726 31836 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:27.476049 31836 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:27.476145 31836 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:27.480131 31836 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:27.480185 31836 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:27.480240 31836 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:27.480299 31836 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:27.480326 31836 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:27.486019 31836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.96.212:50245/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:33297
--ranger_config_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:27.487233 31836 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:27.488297 31836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:27.495496 31842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:27.495501 31841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:27.495537 31844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:27.496245 31836 server_base.cc:1061] running on GCE node
I20260501 02:07:27.496775 31836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:27.497859 31836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:27.499066 31836 hybrid_clock.cc:648] HybridClock initialized: now 1777601247499036 us; error 46 us; skew 500 ppm
May 01 02:07:27 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601247, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:27.503151 31836 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:27.504374 31836 webserver.cc:492] Webserver started at http://127.18.96.254:32801/ using document root <none> and password file <none>
I20260501 02:07:27.505048 31836 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:27.505116 31836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:27.505340 31836 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:07:27 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601247, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:27.554612 31836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "a07479b293ea4b58a9bed0eb4edf6c1a"
format_stamp: "Formatted at 2026-05-01 02:07:27 on dist-test-slave-2j90"
server_key: "9a121b9631a79fe7cdfd69a99974816f"
server_key_iv: "492938a22cd49cee17df23aa7780d088"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:27.555179 31836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "a07479b293ea4b58a9bed0eb4edf6c1a"
format_stamp: "Formatted at 2026-05-01 02:07:27 on dist-test-slave-2j90"
server_key: "9a121b9631a79fe7cdfd69a99974816f"
server_key_iv: "492938a22cd49cee17df23aa7780d088"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:27.558769 31836 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:07:27.575589 31852 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:27.576766 31836 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:27.576876 31836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "a07479b293ea4b58a9bed0eb4edf6c1a"
format_stamp: "Formatted at 2026-05-01 02:07:27 on dist-test-slave-2j90"
server_key: "9a121b9631a79fe7cdfd69a99974816f"
server_key_iv: "492938a22cd49cee17df23aa7780d088"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:27.577015 31836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:27.593465 31836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:27.595767 31836 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:27.595963 31836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:27.603761 31836 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:33297
I20260501 02:07:27.603761 31904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:33297 every 8 connection(s)
I20260501 02:07:27.604753 31836 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:27.607396 31905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:27.609802 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 31836
I20260501 02:07:27.609898 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:07:27.610193 18819 external_mini_cluster.cc:1468] Setting key b03831bc1b8db5cde7d74383b35eab45
I20260501 02:07:27.612602 31905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Bootstrap starting.
I20260501 02:07:27.614971 31905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:27.615680 31905 log.cc:826] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:27.617522 31905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: No bootstrap required, opened a new log
May 01 02:07:27 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:27.620123 31905 raft_consensus.cc:359] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:27.620285 31905 raft_consensus.cc:385] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:27.620369 31905 raft_consensus.cc:740] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07479b293ea4b58a9bed0eb4edf6c1a, State: Initialized, Role: FOLLOWER
I20260501 02:07:27.620726 31905 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:27.620827 31905 raft_consensus.cc:399] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:27.620903 31905 raft_consensus.cc:493] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:27.620987 31905 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:27.621719 31905 raft_consensus.cc:515] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:27.622097 31905 leader_election.cc:304] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07479b293ea4b58a9bed0eb4edf6c1a; no voters:
I20260501 02:07:27.622395 31905 leader_election.cc:290] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:07:27.622462 31910 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:27.622711 31910 raft_consensus.cc:697] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 LEADER]: Becoming Leader. State: Replica: a07479b293ea4b58a9bed0eb4edf6c1a, State: Running, Role: LEADER
I20260501 02:07:27.623018 31910 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:27.623347 31908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:27.611455 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.18.96.254:33297)
0501 02:07:27.611859 (+ 404us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:27.611870 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:27.611905 (+ 35us) server_negotiation.cc:408] Connection header received
0501 02:07:27.612549 (+ 644us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:27.612571 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:27.612898 (+ 327us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:27.613222 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:27.614103 (+ 881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:27.615197 (+ 1094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:27.616105 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:27.616397 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:27.618476 (+ 2079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:27.618491 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:27.618501 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:27.618527 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:27.620929 (+ 2402us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:27.621337 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:27.621343 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:27.621348 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:27.621412 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:27.621659 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:27.621662 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:27.621663 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:27.622059 (+ 396us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:27.622257 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:27.622504 (+ 247us) server_negotiation.cc:300] Negotiation successful
0501 02:07:27.622697 (+ 193us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":99,"threads_started":1}
I20260501 02:07:27.623543 31905 sys_catalog.cc:565] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:27.624537 31912 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07479b293ea4b58a9bed0eb4edf6c1a. Latest consensus state: current_term: 1 leader_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } } }
I20260501 02:07:27.624673 31912 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:27.624966 31911 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } } }
I20260501 02:07:27.625047 31911 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:27.625880 31919 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:27.626256 31905 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260501 02:07:27.628259 31919 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:27.637069 31919 catalog_manager.cc:1357] Generated new cluster ID: 31d49665c05043d2b3e245a89a5572f1
I20260501 02:07:27.637188 31919 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:07:27.660597 31919 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:07:27.661846 31919 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:27.674721 31919 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Generated new TSK 0
I20260501 02:07:27.675634 31919 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:28.376992 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33297
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:28.521654 31964 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:28.521956 31964 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:28.522101 31964 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:28.526580 31964 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:28.526685 31964 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:28.526808 31964 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:07:28.531090 31964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:28.532297 31964 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:28.533161 31964 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:28.539486 31971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:28.539511 31974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:28.539479 31972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:28.540367 31964 server_base.cc:1061] running on GCE node
I20260501 02:07:28.540732 31964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:28.541258 31964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:28.542413 31964 hybrid_clock.cc:648] HybridClock initialized: now 1777601248542388 us; error 41 us; skew 500 ppm
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:28.545099 31964 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:07:28.546128 31964 webserver.cc:492] Webserver started at http://127.18.96.193:37629/ using document root <none> and password file <none>
I20260501 02:07:28.546674 31964 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:28.546744 31964 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:28.546931 31964 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:28.562575 31964 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "b14ec47c95eb4b519535b91a2e6e815f"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "155606692699c4323369630c4fef6b4b"
server_key_iv: "243e21c4d882af5705b4fa26b2b4d255"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:28.563194 31964 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "b14ec47c95eb4b519535b91a2e6e815f"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "155606692699c4323369630c4fef6b4b"
server_key_iv: "243e21c4d882af5705b4fa26b2b4d255"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:28.588477 31964 fs_manager.cc:696] Time spent creating directory manager: real 0.025s user 0.004s sys 0.000s
I20260501 02:07:28.634418 31982 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:28.635782 31964 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.002s sys 0.000s
I20260501 02:07:28.635911 31964 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "b14ec47c95eb4b519535b91a2e6e815f"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "155606692699c4323369630c4fef6b4b"
server_key_iv: "243e21c4d882af5705b4fa26b2b4d255"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:28.636037 31964 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:28.657634 31964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:28.660792 31964 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:28.660970 31964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:28.661567 31964 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:28.662618 31964 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:28.662680 31964 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:28.662734 31964 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:28.662760 31964 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:28.676899 31964 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:40431
I20260501 02:07:28.678041 31964 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:07:28.678552 32097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:40431 every 8 connection(s)
I20260501 02:07:28.684005 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 31964
I20260501 02:07:28.684165 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:07:28.684445 18819 external_mini_cluster.cc:1468] Setting key 3f7c2c430cb3ee181943492665c54161
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:28.710647 32101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:28.682343 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:40599 (local address 127.18.96.254:33297)
0501 02:07:28.690159 (+ 7816us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:28.690164 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:28.690179 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:07:28.690217 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:28.690220 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:28.690278 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:28.690374 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:28.698106 (+ 7732us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:28.698995 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:28.700384 (+ 1389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:28.700585 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:28.706515 (+ 5930us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:28.706537 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:28.706541 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:28.706569 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:28.708406 (+ 1837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:28.709000 (+ 594us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:28.709004 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:28.709005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:28.709058 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:28.709401 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:28.709404 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:28.709406 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:28.709576 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:28.709669 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:28.710340 (+ 671us) server_negotiation.cc:300] Negotiation successful
0501 02:07:28.710463 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7732,"thread_start_us":103,"threads_started":1}
I20260501 02:07:28.712224 32100 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:28.681340 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.193:40599)
0501 02:07:28.685393 (+ 4053us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:28.685435 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:28.688895 (+ 3460us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:28.696411 (+ 7516us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:28.696424 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:28.696776 (+ 352us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:28.697472 (+ 696us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:28.697507 (+ 35us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:28.699151 (+ 1644us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:28.699155 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:28.699795 (+ 640us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:28.699805 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:28.700941 (+ 1136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:28.703408 (+ 2467us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:28.703431 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:28.706335 (+ 2904us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:28.708546 (+ 2211us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:28.708552 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:28.708566 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:28.708879 (+ 313us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:28.709158 (+ 279us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:28.709161 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:28.709163 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:28.709295 (+ 132us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:28.709745 (+ 450us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:28.709750 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:28.710044 (+ 294us) client_negotiation.cc:770] Sending connection context
0501 02:07:28.711085 (+ 1041us) client_negotiation.cc:241] Negotiation successful
0501 02:07:28.711352 (+ 267us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3847,"thread_start_us":136,"threads_started":1}
I20260501 02:07:28.714183 32098 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:28.714450 32098 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:28.715013 32098 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:28.717088 31869 ts_manager.cc:194] Registered new tserver with Master: b14ec47c95eb4b519535b91a2e6e815f (127.18.96.193:40431)
I20260501 02:07:28.718729 31869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:40599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:28.767100 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33297
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260501 02:07:28.921110 32106 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:28.921473 32106 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:28.921573 32106 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:28.926329 32106 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:28.926421 32106 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:28.926548 32106 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:07:28.931915 32106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:28.933002 32106 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:28.934055 32106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:28.941447 32113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:28.941581 32115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:28.941771 32112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:28.942669 32106 server_base.cc:1061] running on GCE node
I20260501 02:07:28.943090 32106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:28.943722 32106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:28.953480 32106 hybrid_clock.cc:648] HybridClock initialized: now 1777601248953460 us; error 49 us; skew 500 ppm
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:28.956988 32106 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:07:28.958382 32106 webserver.cc:492] Webserver started at http://127.18.96.194:34825/ using document root <none> and password file <none>
I20260501 02:07:28.959069 32106 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:28.959131 32106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:28.959358 32106 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:07:28 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:28.975425 32106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "766546e703394d269fa506098977e331"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "a915185586eacb416f18dc016f7adc90"
server_key_iv: "3ad6c2242c3cc339858812e4a6526b5f"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:28.976042 32106 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "766546e703394d269fa506098977e331"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "a915185586eacb416f18dc016f7adc90"
server_key_iv: "3ad6c2242c3cc339858812e4a6526b5f"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:28.986006 32106 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.000s
I20260501 02:07:29.015033 32122 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.018937 32106 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260501 02:07:29.019100 32106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "766546e703394d269fa506098977e331"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "a915185586eacb416f18dc016f7adc90"
server_key_iv: "3ad6c2242c3cc339858812e4a6526b5f"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:29.019268 32106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:29.040081 32106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:29.043502 32106 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:29.043695 32106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:29.044379 32106 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:29.045502 32106 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:29.045570 32106 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.045645 32106 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:29.045681 32106 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.061470 32106 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42699
I20260501 02:07:29.062808 32106 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:07:29.064024 32235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42699 every 8 connection(s)
I20260501 02:07:29.067462 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32106
I20260501 02:07:29.067580 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:07:29.067853 18819 external_mini_cluster.cc:1468] Setting key 833f327facc0e16b4532f62b4550f6ba
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:29.100788 32101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:29.066052 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:48851 (local address 127.18.96.254:33297)
0501 02:07:29.066198 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:29.066204 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:29.078790 (+ 12586us) server_negotiation.cc:408] Connection header received
0501 02:07:29.084988 (+ 6198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:29.084993 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:29.085052 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:29.085103 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:29.086852 (+ 1749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.087692 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:29.088595 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.088789 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:29.093560 (+ 4771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:29.093588 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:29.093591 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:29.093621 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:29.095531 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:29.096853 (+ 1322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:29.096859 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:29.096864 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:29.096921 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:29.099401 (+ 2480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:29.099405 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:29.099407 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:29.099605 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:29.099700 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:29.100428 (+ 728us) server_negotiation.cc:300] Negotiation successful
0501 02:07:29.100586 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:07:29.102555 32238 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:29.066074 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.194:48851)
0501 02:07:29.078611 (+ 12537us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:29.078668 (+ 57us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:29.084707 (+ 6039us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:29.085572 (+ 865us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:29.085584 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:29.085955 (+ 371us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:29.086664 (+ 709us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:29.086680 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.087854 (+ 1174us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:29.087857 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:29.088442 (+ 585us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:29.088452 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.089052 (+ 600us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:29.090310 (+ 1258us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:29.090349 (+ 39us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:29.093340 (+ 2991us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:29.095686 (+ 2346us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:29.095695 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:29.095711 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:29.096035 (+ 324us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:29.097040 (+ 1005us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:29.097045 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:29.097050 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:29.097197 (+ 147us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:29.099767 (+ 2570us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:29.099772 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:29.100123 (+ 351us) client_negotiation.cc:770] Sending connection context
0501 02:07:29.101352 (+ 1229us) client_negotiation.cc:241] Negotiation successful
0501 02:07:29.101597 (+ 245us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12264,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:07:29.105285 32236 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:29.105485 32236 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:29.106158 32236 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
I20260501 02:07:29.107296 31869 ts_manager.cc:194] Registered new tserver with Master: 766546e703394d269fa506098977e331 (127.18.96.194:42699)
I20260501 02:07:29.108067 31869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:48851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:29.185596 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:33297
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:29.348253 32243 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:29.348834 32243 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:29.348958 32243 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:29.353705 32243 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:29.353870 32243 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:29.354058 32243 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:07:29.360014 32243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:29.361464 32243 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:29.362519 32243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:29.370280 32251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:29.370640 32248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:29.370576 32249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:29.371742 32243 server_base.cc:1061] running on GCE node
I20260501 02:07:29.372249 32243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:29.372885 32243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:29.376187 32243 hybrid_clock.cc:648] HybridClock initialized: now 1777601249376161 us; error 37 us; skew 500 ppm
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:29.380071 32243 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:07:29.381486 32243 webserver.cc:492] Webserver started at http://127.18.96.195:43723/ using document root <none> and password file <none>
I20260501 02:07:29.382244 32243 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:29.382350 32243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:29.382622 32243 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:29.401840 32243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "0b83e8c489434dca8db70ebf7a1307a9"
format_stamp: "Formatted at 2026-05-01 02:07:29 on dist-test-slave-2j90"
server_key: "2e82d9078b3bb8316078b27c5015c9e7"
server_key_iv: "d20feace1c5faa1167401cb67f7ea85c"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:29.403877 32243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "0b83e8c489434dca8db70ebf7a1307a9"
format_stamp: "Formatted at 2026-05-01 02:07:29 on dist-test-slave-2j90"
server_key: "2e82d9078b3bb8316078b27c5015c9e7"
server_key_iv: "d20feace1c5faa1167401cb67f7ea85c"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:29.410825 32243 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601248, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:29.426458 32259 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.427784 32243 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260501 02:07:29.427893 32243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "0b83e8c489434dca8db70ebf7a1307a9"
format_stamp: "Formatted at 2026-05-01 02:07:29 on dist-test-slave-2j90"
server_key: "2e82d9078b3bb8316078b27c5015c9e7"
server_key_iv: "d20feace1c5faa1167401cb67f7ea85c"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:29.428000 32243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:29.468499 32243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:29.472841 32243 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:29.474351 32243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:29.475451 32243 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:29.484549 32243 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:29.484673 32243 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20260501 02:07:29.484776 32243 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:29.484840 32243 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.496182 32243 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:37809
I20260501 02:07:29.497941 32243 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:07:29.499018 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32243
I20260501 02:07:29.499155 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:07:29.499439 18819 external_mini_cluster.cc:1468] Setting key 04a8f32da111921b4a5298567a3fe3cd
I20260501 02:07:29.507632 32372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:37809 every 8 connection(s)
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:29.536099 32101 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:29.510068 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:43339 (local address 127.18.96.254:33297)
0501 02:07:29.510218 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:29.510222 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:29.511645 (+ 1423us) server_negotiation.cc:408] Connection header received
0501 02:07:29.512722 (+ 1077us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:29.512725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:29.512790 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:29.512835 (+ 45us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:29.514421 (+ 1586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.515212 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:29.516110 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.516317 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:29.519066 (+ 2749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:29.519081 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:29.519084 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:29.519110 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:29.520781 (+ 1671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:29.534300 (+ 13519us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:29.534304 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:29.534307 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:29.534371 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:29.534758 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:29.534762 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:29.534764 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:29.534950 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:29.535056 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:29.535777 (+ 721us) server_negotiation.cc:300] Negotiation successful
0501 02:07:29.535911 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:07:29.539006 32375 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:29.510449 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.195:43339)
0501 02:07:29.511519 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:29.511558 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:29.512507 (+ 949us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:29.513207 (+ 700us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:29.513217 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:29.513558 (+ 341us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:29.514258 (+ 700us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:29.514274 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.515390 (+ 1116us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:29.515393 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:29.515977 (+ 584us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:29.515986 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:29.516541 (+ 555us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:29.517175 (+ 634us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:29.517196 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:29.518901 (+ 1705us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:29.520955 (+ 2054us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:29.520972 (+ 17us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:29.533675 (+ 12703us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:29.534140 (+ 465us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:29.534502 (+ 362us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:29.534506 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:29.534509 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:29.534647 (+ 138us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:29.535141 (+ 494us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:29.535147 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:29.535469 (+ 322us) client_negotiation.cc:770] Sending connection context
0501 02:07:29.537645 (+ 2176us) client_negotiation.cc:241] Negotiation successful
0501 02:07:29.537895 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":127,"threads_started":1}
I20260501 02:07:29.540174 32373 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:29.540422 32373 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:29.541003 32373 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:29.542114 31869 ts_manager.cc:194] Registered new tserver with Master: 0b83e8c489434dca8db70ebf7a1307a9 (127.18.96.195:37809)
I20260501 02:07:29.542855 31869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:43339
I20260501 02:07:29.549265 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:07:29.549913 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 31964
I20260501 02:07:29.558617 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32106
I20260501 02:07:29.567729 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32243
I20260501 02:07:29.573565 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 31836
I20260501 02:07:29.581689 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33297
--webserver_interface=127.18.96.254
--webserver_port=32801
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:33297
--ranger_config_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:29.703511 32376 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:29.703761 32376 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:29.703851 32376 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:29.707374 32376 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:29.707465 32376 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:29.707522 32376 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:29.707576 32376 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:29.707621 32376 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:29.712297 32376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:33297
--ranger_config_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:33297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=32801
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:29.713372 32376 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:29.714273 32376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:29.719894 32381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:29.719902 32382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:29.720027 32384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:29.720227 32376 server_base.cc:1061] running on GCE node
I20260501 02:07:29.720860 32376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:29.721751 32376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:29.722935 32376 hybrid_clock.cc:648] HybridClock initialized: now 1777601249722912 us; error 47 us; skew 500 ppm
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:29.725804 32376 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:29.726856 32376 webserver.cc:492] Webserver started at http://127.18.96.254:32801/ using document root <none> and password file <none>
I20260501 02:07:29.727429 32376 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:29.727519 32376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:07:29 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601249, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:29.803155 32376 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:07:29.805272 32392 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:29.806406 32376 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:07:29.806545 32376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "a07479b293ea4b58a9bed0eb4edf6c1a"
format_stamp: "Formatted at 2026-05-01 02:07:27 on dist-test-slave-2j90"
server_key: "9a121b9631a79fe7cdfd69a99974816f"
server_key_iv: "492938a22cd49cee17df23aa7780d088"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:29.806939 32376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:29.869179 32376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:29.871591 32376 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:29.871732 32376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:29.879174 32376 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:33297
I20260501 02:07:29.879174 32444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:33297 every 8 connection(s)
I20260501 02:07:29.880267 32376 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:29.884130 32445 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:07:29.885633 32445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Bootstrap starting.
I20260501 02:07:29.889703 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32376
I20260501 02:07:29.890206 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:40431
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:33297
--webserver_port=37629
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:29.901369 32445 log.cc:826] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:29.907500 32445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:07:29.907868 32445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Bootstrap complete.
I20260501 02:07:29.911723 32445 raft_consensus.cc:359] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:29.912061 32445 raft_consensus.cc:740] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07479b293ea4b58a9bed0eb4edf6c1a, State: Initialized, Role: FOLLOWER
I20260501 02:07:29.912537 32445 consensus_queue.cc:260] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:29.912644 32445 raft_consensus.cc:399] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:29.912725 32445 raft_consensus.cc:493] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:29.912853 32445 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:07:29.914397 32445 raft_consensus.cc:515] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:29.914705 32445 leader_election.cc:304] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a07479b293ea4b58a9bed0eb4edf6c1a; no voters:
I20260501 02:07:29.915012 32445 leader_election.cc:290] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 02:07:29.915093 32449 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 2 FOLLOWER]: Leader election won for term 2
I20260501 02:07:29.915314 32449 raft_consensus.cc:697] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [term 2 LEADER]: Becoming Leader. State: Replica: a07479b293ea4b58a9bed0eb4edf6c1a, State: Running, Role: LEADER
I20260501 02:07:29.915648 32449 consensus_queue.cc:237] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } }
I20260501 02:07:29.916062 32445 sys_catalog.cc:565] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:29.916462 32449 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a07479b293ea4b58a9bed0eb4edf6c1a. Latest consensus state: current_term: 2 leader_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } } }
I20260501 02:07:29.916566 32449 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:29.917017 32450 sys_catalog.cc:455] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a07479b293ea4b58a9bed0eb4edf6c1a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 33297 } } }
I20260501 02:07:29.917165 32450 sys_catalog.cc:458] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:29.917973 32453 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:29.918462 32445 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260501 02:07:29.924127 32453 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:29.925088 32453 catalog_manager.cc:1269] Loaded cluster ID: 31d49665c05043d2b3e245a89a5572f1
I20260501 02:07:29.925194 32453 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:07:29.928006 32453 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:29.928920 32453 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Loaded TSK: 0
I20260501 02:07:29.930060 32453 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:07:30.022742 32447 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:30.023085 32447 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:30.023191 32447 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:30.027741 32447 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:30.027881 32447 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:30.028053 32447 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:07:30.033885 32447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:40431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=37629
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:30.035210 32447 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:30.036319 32447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:30.043912 32495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:30.044405 32493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.044902 32447 server_base.cc:1061] running on GCE node
W20260501 02:07:30.045158 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.045629 32447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:30.046310 32447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:30.053707 32447 hybrid_clock.cc:648] HybridClock initialized: now 1777601250053678 us; error 43 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:30.057694 32447 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:07:30.059140 32447 webserver.cc:492] Webserver started at http://127.18.96.193:37629/ using document root <none> and password file <none>
I20260501 02:07:30.059906 32447 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:30.060045 32447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:30.095415 32447 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260501 02:07:30.097872 32503 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.099189 32447 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:07:30.099311 32447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "b14ec47c95eb4b519535b91a2e6e815f"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "155606692699c4323369630c4fef6b4b"
server_key_iv: "243e21c4d882af5705b4fa26b2b4d255"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:30.099787 32447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:30.120584 32447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:30.123865 32447 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:30.124109 32447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:30.124786 32447 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:30.125896 32447 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:30.126042 32447 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.126173 32447 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:30.126296 32447 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.139150 32447 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:40431
I20260501 02:07:30.140236 32447 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:07:30.139168 32616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:40431 every 8 connection(s)
I20260501 02:07:30.150331 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32447
I20260501 02:07:30.150839 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:42699
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:33297
--webserver_port=34825
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:30.166924 32620 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.143731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.193:50483)
0501 02:07:30.144222 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:30.144268 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:30.148300 (+ 4032us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:30.149907 (+ 1607us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:30.149916 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:30.152773 (+ 2857us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:30.153365 (+ 592us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.153380 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.155758 (+ 2378us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.155762 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:30.156347 (+ 585us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.156357 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.156592 (+ 235us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.157366 (+ 774us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:30.157390 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:30.159038 (+ 1648us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:30.163806 (+ 4768us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.163812 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.163823 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.164113 (+ 290us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.164452 (+ 339us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.164455 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.164458 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.164573 (+ 115us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.165459 (+ 886us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:30.165466 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:30.165724 (+ 258us) client_negotiation.cc:770] Sending connection context
0501 02:07:30.165912 (+ 188us) client_negotiation.cc:241] Negotiation successful
0501 02:07:30.166159 (+ 247us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260501 02:07:30.167474 32619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.143310 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:50483 (local address 127.18.96.254:33297)
0501 02:07:30.143867 (+ 557us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:30.143883 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:30.144414 (+ 531us) server_negotiation.cc:408] Connection header received
0501 02:07:30.148889 (+ 4475us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:30.148902 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:30.149200 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.149496 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:30.153566 (+ 4070us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.155596 (+ 2030us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.156886 (+ 1290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.157140 (+ 254us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.159221 (+ 2081us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.159263 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:30.159275 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:30.159308 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:30.163674 (+ 4366us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.164262 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.164269 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.164274 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.164349 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.164674 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.164678 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.164680 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.165250 (+ 570us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.165398 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:30.166239 (+ 841us) server_negotiation.cc:300] Negotiation successful
0501 02:07:30.166465 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":121,"threads_started":1}
I20260501 02:07:30.170131 32617 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:30.170384 32617 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.170912 32617 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.172978 32617 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:33297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260501 02:07:30.173084 32617 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.173267 32617 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:30.174139 32617 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.174314 32617 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.175045 32617 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:07:30.295907 32621 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:30.296196 32621 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:30.296301 32621 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:30.300966 32621 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:30.301127 32621 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:30.301322 32621 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:07:30.306826 32621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:42699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=34825
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:30.307895 32621 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:30.308729 32621 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:30.320201 32626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.320375 32621 server_base.cc:1061] running on GCE node
W20260501 02:07:30.320658 32629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:30.320837 32627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.321275 32621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:30.321986 32621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:30.323184 32621 hybrid_clock.cc:648] HybridClock initialized: now 1777601250323166 us; error 32 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:30.326961 32621 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:07:30.328181 32621 webserver.cc:492] Webserver started at http://127.18.96.194:34825/ using document root <none> and password file <none>
I20260501 02:07:30.328902 32621 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:30.328987 32621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:30.350787 32621 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260501 02:07:30.353657 32637 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.354852 32621 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:07:30.354990 32621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "766546e703394d269fa506098977e331"
format_stamp: "Formatted at 2026-05-01 02:07:28 on dist-test-slave-2j90"
server_key: "a915185586eacb416f18dc016f7adc90"
server_key_iv: "3ad6c2242c3cc339858812e4a6526b5f"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:30.355405 32621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:30.385569 32621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:30.388581 32621 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:30.388793 32621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:30.389408 32621 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:30.390552 32621 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:30.390667 32621 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.390784 32621 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:30.390853 32621 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.404961 32621 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42699
I20260501 02:07:30.404979 32750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42699 every 8 connection(s)
I20260501 02:07:30.406363 32621 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:07:30.408555 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32621
I20260501 02:07:30.409047 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.96.212:50245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:37809
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:33297
--webserver_port=43723
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:30.423956 32619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.408234 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:33343 (local address 127.18.96.254:33297)
0501 02:07:30.410749 (+ 2515us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:30.410754 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:30.410772 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:07:30.411421 (+ 649us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:30.411425 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:30.411483 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.411567 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:30.412971 (+ 1404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.413471 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.414200 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.414405 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.418342 (+ 3937us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.418368 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:30.418372 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:30.418402 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:30.420124 (+ 1722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.420663 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.420667 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.420669 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.420718 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.421070 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.421078 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.421082 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.422719 (+ 1637us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.422845 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:30.423563 (+ 718us) server_negotiation.cc:300] Negotiation successful
0501 02:07:30.423706 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2412}
I20260501 02:07:30.425410 32753 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.409036 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.194:33343)
0501 02:07:30.410107 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:30.410151 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:30.411207 (+ 1056us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:30.411902 (+ 695us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:30.411914 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:30.412287 (+ 373us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:30.412753 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.412763 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.413601 (+ 838us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.413605 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:30.414082 (+ 477us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.414090 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.414325 (+ 235us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.416047 (+ 1722us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:30.416074 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:30.418139 (+ 2065us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:30.420258 (+ 2119us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.420263 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.420284 (+ 21us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.420555 (+ 271us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.420826 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.420831 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.420835 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.420963 (+ 128us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.422912 (+ 1949us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:30.422919 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:30.423214 (+ 295us) client_negotiation.cc:770] Sending connection context
0501 02:07:30.424253 (+ 1039us) client_negotiation.cc:241] Negotiation successful
0501 02:07:30.424483 (+ 230us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":123,"threads_started":1}
I20260501 02:07:30.426829 32751 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:30.435539 32751 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.436247 32751 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.438030 32751 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:33297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260501 02:07:30.438205 32751 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.439042 32751 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:30.439972 32751 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.440193 32751 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.442135 32751 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:07:30.566447 32754 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:30.566818 32754 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:30.566959 32754 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:30.571977 32754 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:30.572117 32754 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:30.572294 32754 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:07:30.578389 32754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.96.212:50245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:37809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=43723
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:33297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:30.579746 32754 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:30.580649 32754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:30.588320 32763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:30.588366 32765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.588807 32754 server_base.cc:1061] running on GCE node
W20260501 02:07:30.589157 32762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:30.589577 32754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:30.590225 32754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:30.591382 32754 hybrid_clock.cc:648] HybridClock initialized: now 1777601250591362 us; error 29 us; skew 500 ppm
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:30.594376 32754 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:07:30.595506 32754 webserver.cc:492] Webserver started at http://127.18.96.195:43723/ using document root <none> and password file <none>
I20260501 02:07:30.596053 32754 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:30.596141 32754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:30.623512 32754 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.002s sys 0.000s
I20260501 02:07:30.630702 305 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.632053 32754 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260501 02:07:30.632175 32754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "0b83e8c489434dca8db70ebf7a1307a9"
format_stamp: "Formatted at 2026-05-01 02:07:29 on dist-test-slave-2j90"
server_key: "2e82d9078b3bb8316078b27c5015c9e7"
server_key_iv: "d20feace1c5faa1167401cb67f7ea85c"
server_key_version: "kuduclusterkey@0"
I20260501 02:07:30.632740 32754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:30.650204 32754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:30.653338 32754 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:30.653561 32754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:30.654258 32754 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:30.655341 32754 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:30.655406 32754 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.655460 32754 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:30.655499 32754 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:30.667973 32754 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:37809
I20260501 02:07:30.669257 32754 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:07:30.674621 418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:37809 every 8 connection(s)
I20260501 02:07:30.674748 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 32754
May 01 02:07:30 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601250, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:30.708967 421 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.677131 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:33297 (local address 127.18.96.195:39513)
0501 02:07:30.683701 (+ 6570us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:30.683752 (+ 51us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:30.684851 (+ 1099us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:30.698260 (+ 13409us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:30.698285 (+ 25us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:30.698684 (+ 399us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:30.699417 (+ 733us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.699438 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.700618 (+ 1180us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.700622 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:30.701221 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:30.701229 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.701821 (+ 592us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.702501 (+ 680us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:30.702524 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:30.704194 (+ 1670us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:30.706315 (+ 2121us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.706321 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.706335 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.706644 (+ 309us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.706899 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:30.706902 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:30.706904 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:30.707014 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:30.707415 (+ 401us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:30.707421 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:30.707698 (+ 277us) client_negotiation.cc:770] Sending connection context
0501 02:07:30.707895 (+ 197us) client_negotiation.cc:241] Negotiation successful
0501 02:07:30.708121 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6352,"thread_start_us":134,"threads_started":1}
I20260501 02:07:30.709707 32619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.678144 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:39513 (local address 127.18.96.254:33297)
0501 02:07:30.678295 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:30.678300 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:30.683863 (+ 5563us) server_negotiation.cc:408] Connection header received
0501 02:07:30.685079 (+ 1216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:30.685084 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:30.685143 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.685236 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:30.699639 (+ 14403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.700452 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.701376 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.701571 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.704373 (+ 2802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.704396 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:30.704399 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:30.704426 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:30.706174 (+ 1748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.706759 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.706762 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.706764 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.706810 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.707120 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.707124 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.707125 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.707280 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.707365 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:30.709411 (+ 2046us) server_negotiation.cc:300] Negotiation successful
0501 02:07:30.709561 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:07:30.710669 419 heartbeater.cc:344] Connected to a master server at 127.18.96.254:33297
I20260501 02:07:30.710937 419 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.711519 419 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.713070 419 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:33297 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260501 02:07:30.713188 419 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.713389 419 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:30.714198 419 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:30.714406 419 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
W20260501 02:07:30.715139 419 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260501 02:07:30.731170 423 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:30.678417 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.18.96.254:33297)
0501 02:07:30.683380 (+ 4963us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:30.683385 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:30.692785 (+ 9400us) server_negotiation.cc:408] Connection header received
0501 02:07:30.692843 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:30.692847 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:30.692904 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:30.692995 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:30.701437 (+ 8442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.704147 (+ 2710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:30.706692 (+ 2545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:30.706898 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:30.708157 (+ 1259us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:30.708179 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:30.708182 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:30.708209 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:30.710923 (+ 2714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.718553 (+ 7630us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.718558 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.718560 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.718615 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:30.730066 (+ 11451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:30.730072 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:30.730074 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:30.730312 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:30.730553 (+ 241us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:30.730830 (+ 277us) server_negotiation.cc:300] Negotiation successful
0501 02:07:30.730984 (+ 154us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4865,"thread_start_us":75,"threads_started":1}
I20260501 02:07:31.175268 32617 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:31.175647 32617 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:31.176915 32405 ts_manager.cc:194] Registered new tserver with Master: b14ec47c95eb4b519535b91a2e6e815f (127.18.96.193:40431)
I20260501 02:07:31.178767 32405 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:50483
May 01 02:07:31 dist-test-slave-2j90 krb5kdc[31112](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:31.442386 32751 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:31.442741 32751 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:31.443513 32405 ts_manager.cc:194] Registered new tserver with Master: 766546e703394d269fa506098977e331 (127.18.96.194:42699)
I20260501 02:07:31.444291 32405 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:33343
May 01 02:07:31 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
I20260501 02:07:31.715366 419 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:31.715730 419 heartbeater.cc:507] Master 127.18.96.254:33297 requested a full tablet report, sending...
I20260501 02:07:31.716480 32405 ts_manager.cc:194] Registered new tserver with Master: 0b83e8c489434dca8db70ebf7a1307a9 (127.18.96.195:37809)
I20260501 02:07:31.717125 32405 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:39513
I20260501 02:07:31.724985 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:07:31.740626 440 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.730702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.18.96.254:33297)
0501 02:07:31.731283 (+ 581us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.731288 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.731652 (+ 364us) server_negotiation.cc:408] Connection header received
0501 02:07:31.731695 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.731698 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.731751 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.731866 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:31.733059 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.733888 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.735291 (+ 1403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.735521 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.736813 (+ 1292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:31.736833 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:31.736836 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:31.736863 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:31.738674 (+ 1811us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.739135 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.739139 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:31.739141 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:31.739191 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:31.739451 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:31.739455 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:31.739457 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:31.739626 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:31.739740 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:31.740326 (+ 586us) server_negotiation.cc:300] Negotiation successful
0501 02:07:31.740437 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":85,"threads_started":1}
I20260501 02:07:31.744225 32405 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:07:31.747929 32405 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:07:31.779553 447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.767889 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42699 (local address 127.0.0.1:42622)
0501 02:07:31.774265 (+ 6376us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.774300 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:31.774443 (+ 143us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.775768 (+ 1325us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:31.775773 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:31.775791 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:31.776050 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.776058 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.777479 (+ 1421us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.777482 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:31.779077 (+ 1595us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.779085 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.779188 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.779235 (+ 47us) client_negotiation.cc:770] Sending connection context
0501 02:07:31.779334 (+ 99us) client_negotiation.cc:241] Negotiation successful
0501 02:07:31.779403 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6201,"thread_start_us":103,"threads_started":1}
I20260501 02:07:31.780045 450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.770150 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.18.96.193:40431)
0501 02:07:31.774835 (+ 4685us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.774841 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.775289 (+ 448us) server_negotiation.cc:408] Connection header received
0501 02:07:31.775788 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.775793 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.775945 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.776034 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:31.776599 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.777688 (+ 1089us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.778822 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.779350 (+ 528us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.779450 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:31.779849 (+ 399us) server_negotiation.cc:300] Negotiation successful
0501 02:07:31.779922 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4581,"thread_start_us":128,"threads_started":1}
I20260501 02:07:31.779753 451 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.770033 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40431 (local address 127.0.0.1:34796)
0501 02:07:31.775205 (+ 5172us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.775218 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:31.775619 (+ 401us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.776204 (+ 585us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:31.776207 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:31.776226 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:31.776466 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.776472 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.777807 (+ 1335us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.777811 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:31.778687 (+ 876us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.778698 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.779518 (+ 820us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.779530 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:07:31.779596 (+ 66us) client_negotiation.cc:241] Negotiation successful
0501 02:07:31.779646 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5101,"thread_start_us":96,"threads_started":1}
I20260501 02:07:31.783447 32541 tablet_service.cc:1511] Processing CreateTablet for tablet c149f7fe30f841a788b4c97c46b06555 (DEFAULT_TABLE table=test-table [id=127647a7fabd4830b91cbcc75ce6240b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:31.784546 32541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c149f7fe30f841a788b4c97c46b06555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:31.789563 448 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.767990 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.18.96.194:42699)
0501 02:07:31.775025 (+ 7035us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.775031 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.775052 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:07:31.775218 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.775224 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.775443 (+ 219us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.775561 (+ 118us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:31.776192 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.777358 (+ 1166us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.779971 (+ 2613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.789180 (+ 9209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.789294 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:31.789360 (+ 66us) server_negotiation.cc:300] Negotiation successful
0501 02:07:31.789431 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6924,"thread_start_us":72,"threads_started":1}
I20260501 02:07:31.793560 449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.770341 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:37809 (local address 127.0.0.1:42618)
0501 02:07:31.775413 (+ 5072us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.775425 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:31.775512 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.790588 (+ 15076us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:31.790592 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:31.790614 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:31.790874 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.790882 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.792336 (+ 1454us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.792340 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:31.793176 (+ 836us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.793186 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.793292 (+ 106us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.793308 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:07:31.793345 (+ 37us) client_negotiation.cc:241] Negotiation successful
0501 02:07:31.793396 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5017,"thread_start_us":75,"threads_started":1}
I20260501 02:07:31.794749 452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.770689 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.18.96.195:37809)
0501 02:07:31.790147 (+ 19458us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.790153 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.790174 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:07:31.790247 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.790251 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.790402 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.791032 (+ 630us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:31.791170 (+ 138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.792208 (+ 1038us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.793844 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.794385 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.794478 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:31.794548 (+ 70us) server_negotiation.cc:300] Negotiation successful
0501 02:07:31.794623 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19345,"thread_start_us":93,"threads_started":1}
I20260501 02:07:31.794880 32685 tablet_service.cc:1511] Processing CreateTablet for tablet c149f7fe30f841a788b4c97c46b06555 (DEFAULT_TABLE table=test-table [id=127647a7fabd4830b91cbcc75ce6240b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:31.795804 32685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c149f7fe30f841a788b4c97c46b06555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:31.799294 353 tablet_service.cc:1511] Processing CreateTablet for tablet c149f7fe30f841a788b4c97c46b06555 (DEFAULT_TABLE table=test-table [id=127647a7fabd4830b91cbcc75ce6240b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:31.800093 353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c149f7fe30f841a788b4c97c46b06555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:31.812517 455 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Bootstrap starting.
I20260501 02:07:31.814162 456 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Bootstrap starting.
I20260501 02:07:31.815368 455 tablet_bootstrap.cc:654] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:31.815532 454 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Bootstrap starting.
I20260501 02:07:31.816151 456 tablet_bootstrap.cc:654] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:31.817054 456 log.cc:826] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:31.818503 454 tablet_bootstrap.cc:654] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:31.818641 455 log.cc:826] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:31.819329 456 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: No bootstrap required, opened a new log
I20260501 02:07:31.819507 456 ts_tablet_manager.cc:1403] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:07:31.819849 454 log.cc:826] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:31.823273 456 raft_consensus.cc:359] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.823510 456 raft_consensus.cc:385] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:31.823573 456 raft_consensus.cc:740] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14ec47c95eb4b519535b91a2e6e815f, State: Initialized, Role: FOLLOWER
I20260501 02:07:31.824023 456 consensus_queue.cc:260] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.824908 456 ts_tablet_manager.cc:1434] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:07:31.825250 32617 heartbeater.cc:499] Master 127.18.96.254:33297 was elected leader, sending a full tablet report...
I20260501 02:07:31.827147 455 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: No bootstrap required, opened a new log
I20260501 02:07:31.827355 455 ts_tablet_manager.cc:1403] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260501 02:07:31.830701 455 raft_consensus.cc:359] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.830892 455 raft_consensus.cc:385] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:31.830943 455 raft_consensus.cc:740] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766546e703394d269fa506098977e331, State: Initialized, Role: FOLLOWER
I20260501 02:07:31.832464 454 tablet_bootstrap.cc:492] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: No bootstrap required, opened a new log
I20260501 02:07:31.831461 455 consensus_queue.cc:260] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.832684 454 ts_tablet_manager.cc:1403] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260501 02:07:31.833461 455 ts_tablet_manager.cc:1434] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:07:31.833840 32751 heartbeater.cc:499] Master 127.18.96.254:33297 was elected leader, sending a full tablet report...
I20260501 02:07:31.837728 454 raft_consensus.cc:359] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.838064 454 raft_consensus.cc:385] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:31.838191 454 raft_consensus.cc:740] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b83e8c489434dca8db70ebf7a1307a9, State: Initialized, Role: FOLLOWER
I20260501 02:07:31.838802 454 consensus_queue.cc:260] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.843065 454 ts_tablet_manager.cc:1434] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260501 02:07:31.843443 419 heartbeater.cc:499] Master 127.18.96.254:33297 was elected leader, sending a full tablet report...
May 01 02:07:31 dist-test-slave-2j90 krb5kdc[31112](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777601251, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for HTTP/127.18.96.212@KRBTEST.COM
W20260501 02:07:31.895252 32618 tablet.cc:2404] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:07:31.908427 32752 tablet.cc:2404] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:07:31.934176 420 tablet.cc:2404] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:07:31.977819 461 raft_consensus.cc:493] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:07:31.978071 461 raft_consensus.cc:515] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.979454 461 leader_election.cc:290] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14ec47c95eb4b519535b91a2e6e815f (127.18.96.193:40431), 0b83e8c489434dca8db70ebf7a1307a9 (127.18.96.195:37809)
I20260501 02:07:31.993564 450 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.979979 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34417 (local address 127.18.96.193:40431)
0501 02:07:31.980733 (+ 754us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.980737 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.989463 (+ 8726us) server_negotiation.cc:408] Connection header received
0501 02:07:31.989714 (+ 251us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.989718 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.989776 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.989825 (+ 49us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:31.990504 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.991285 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.992564 (+ 1279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.993069 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.993111 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:31.993355 (+ 244us) server_negotiation.cc:300] Negotiation successful
0501 02:07:31.993421 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650}
I20260501 02:07:31.994262 464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.979920 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40431 (local address 127.18.96.194:34417)
0501 02:07:31.989329 (+ 9409us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.989351 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:31.989569 (+ 218us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.989988 (+ 419us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:31.989991 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:31.990011 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:31.990349 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.990359 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.991433 (+ 1074us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:31.991436 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:31.992421 (+ 985us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.992431 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.993183 (+ 752us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:31.993199 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:07:31.994036 (+ 837us) client_negotiation.cc:241] Negotiation successful
0501 02:07:31.994103 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9272,"thread_start_us":128,"threads_started":1}
I20260501 02:07:31.995136 32571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c149f7fe30f841a788b4c97c46b06555" candidate_uuid: "766546e703394d269fa506098977e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ec47c95eb4b519535b91a2e6e815f" is_pre_election: true
I20260501 02:07:31.995474 32571 raft_consensus.cc:2468] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766546e703394d269fa506098977e331 in term 0.
I20260501 02:07:31.996138 32641 leader_election.cc:304] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 766546e703394d269fa506098977e331, b14ec47c95eb4b519535b91a2e6e815f; no voters:
I20260501 02:07:31.996475 461 raft_consensus.cc:2804] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:07:31.996541 461 raft_consensus.cc:493] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:07:31.996582 461 raft_consensus.cc:3060] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:31.997826 461 raft_consensus.cc:515] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:31.998255 461 leader_election.cc:290] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [CANDIDATE]: Term 1 election: Requested vote from peers b14ec47c95eb4b519535b91a2e6e815f (127.18.96.193:40431), 0b83e8c489434dca8db70ebf7a1307a9 (127.18.96.195:37809)
I20260501 02:07:31.998931 32571 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c149f7fe30f841a788b4c97c46b06555" candidate_uuid: "766546e703394d269fa506098977e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ec47c95eb4b519535b91a2e6e815f"
I20260501 02:07:31.999049 32571 raft_consensus.cc:3060] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:32.000358 32571 raft_consensus.cc:2468] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766546e703394d269fa506098977e331 in term 1.
I20260501 02:07:32.000747 32641 leader_election.cc:304] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 766546e703394d269fa506098977e331, b14ec47c95eb4b519535b91a2e6e815f; no voters:
I20260501 02:07:32.000986 461 raft_consensus.cc:2804] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:32.001219 461 raft_consensus.cc:697] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 1 LEADER]: Becoming Leader. State: Replica: 766546e703394d269fa506098977e331, State: Running, Role: LEADER
I20260501 02:07:32.001564 461 consensus_queue.cc:237] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } }
I20260501 02:07:32.002023 452 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.980496 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34107 (local address 127.18.96.195:37809)
0501 02:07:31.980875 (+ 379us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:31.980880 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:31.998305 (+ 17425us) server_negotiation.cc:408] Connection header received
0501 02:07:31.998568 (+ 263us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:31.998572 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:31.998625 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:31.998672 (+ 47us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:31.999267 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:31.999955 (+ 688us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.001093 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.001557 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.001594 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:32.001796 (+ 202us) server_negotiation.cc:300] Negotiation successful
0501 02:07:32.001856 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260501 02:07:32.002409 465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:31.980364 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:37809 (local address 127.18.96.194:34107)
0501 02:07:31.998186 (+ 17822us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:31.998204 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:31.998398 (+ 194us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:31.998806 (+ 408us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:31.998809 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:31.998826 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:31.999118 (+ 292us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:31.999125 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.000097 (+ 972us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.000100 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:32.000936 (+ 836us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:32.000964 (+ 28us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.001656 (+ 692us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.001670 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:07:32.002213 (+ 543us) client_negotiation.cc:241] Negotiation successful
0501 02:07:32.002288 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17731,"thread_start_us":132,"threads_started":1}
I20260501 02:07:32.003271 354 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c149f7fe30f841a788b4c97c46b06555" candidate_uuid: "766546e703394d269fa506098977e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b83e8c489434dca8db70ebf7a1307a9" is_pre_election: true
I20260501 02:07:32.003527 354 raft_consensus.cc:2468] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766546e703394d269fa506098977e331 in term 0.
I20260501 02:07:32.003865 355 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c149f7fe30f841a788b4c97c46b06555" candidate_uuid: "766546e703394d269fa506098977e331" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b83e8c489434dca8db70ebf7a1307a9"
I20260501 02:07:32.003965 355 raft_consensus.cc:3060] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:32.005206 355 raft_consensus.cc:2468] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766546e703394d269fa506098977e331 in term 1.
I20260501 02:07:32.008718 32405 catalog_manager.cc:5671] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 reported cstate change: term changed from 0 to 1, leader changed from <none> to 766546e703394d269fa506098977e331 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "766546e703394d269fa506098977e331" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766546e703394d269fa506098977e331" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 } health_report { overall_health: UNKNOWN } } }
I20260501 02:07:32.074754 448 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:32.070711 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.18.96.194:42699)
0501 02:07:32.070856 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:32.070860 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:32.070875 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:07:32.070963 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:32.070966 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:32.071026 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:32.071114 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:07:32.071616 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.072358 (+ 742us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:32.073345 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:32.073547 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:32.073638 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:32.074095 (+ 457us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:32.074188 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:32.074571 (+ 383us) server_negotiation.cc:300] Negotiation successful
0501 02:07:32.074617 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260501 02:07:32.094419 355 raft_consensus.cc:1275] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 1 FOLLOWER]: Refusing update from remote peer 766546e703394d269fa506098977e331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:32.095338 32571 raft_consensus.cc:1275] T c149f7fe30f841a788b4c97c46b06555 P b14ec47c95eb4b519535b91a2e6e815f [term 1 FOLLOWER]: Refusing update from remote peer 766546e703394d269fa506098977e331: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:32.096144 466 consensus_queue.cc:1048] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b83e8c489434dca8db70ebf7a1307a9" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:32.096611 466 consensus_queue.cc:1048] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14ec47c95eb4b519535b91a2e6e815f" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:32.105439 469 mvcc.cc:204] Tried to move back new op lower bound from 7281054728568463360 to 7281054728217751552. Current Snapshot: MvccSnapshot[applied={T|T < 7281054728568463360}]
I20260501 02:07:32.107788 470 mvcc.cc:204] Tried to move back new op lower bound from 7281054728568463360 to 7281054728217751552. Current Snapshot: MvccSnapshot[applied={T|T < 7281054728568463360}]
I20260501 02:07:32.112040 472 mvcc.cc:204] Tried to move back new op lower bound from 7281054728568463360 to 7281054728217751552. Current Snapshot: MvccSnapshot[applied={T|T < 7281054728568463360}]
I20260501 02:07:32.116767 32409 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:32.116953 32409 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:32.120359 32409 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a07479b293ea4b58a9bed0eb4edf6c1a: Sending DeleteTablet for 3 replicas of tablet c149f7fe30f841a788b4c97c46b06555
I20260501 02:07:32.121737 32685 tablet_service.cc:1558] Processing DeleteTablet for tablet c149f7fe30f841a788b4c97c46b06555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42622
I20260501 02:07:32.122256 32541 tablet_service.cc:1558] Processing DeleteTablet for tablet c149f7fe30f841a788b4c97c46b06555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:34796
I20260501 02:07:32.122695 353 tablet_service.cc:1558] Processing DeleteTablet for tablet c149f7fe30f841a788b4c97c46b06555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:32 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:42618
I20260501 02:07:32.124673 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32447
I20260501 02:07:32.124928 479 tablet_replica.cc:333] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: stopping tablet replica
I20260501 02:07:32.125237 479 raft_consensus.cc:2243] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:07:32.126336 479 raft_consensus.cc:2272] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:32.127800 481 tablet_replica.cc:333] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: stopping tablet replica
I20260501 02:07:32.128050 481 raft_consensus.cc:2243] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:32.128391 481 raft_consensus.cc:2272] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:32.129769 479 ts_tablet_manager.cc:1916] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:32.129769 481 ts_tablet_manager.cc:1916] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:32.132438 479 ts_tablet_manager.cc:1929] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:32.132800 481 ts_tablet_manager.cc:1929] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:32.132891 481 log.cc:1199] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/c149f7fe30f841a788b4c97c46b06555
I20260501 02:07:32.133235 481 ts_tablet_manager.cc:1950] T c149f7fe30f841a788b4c97c46b06555 P 0b83e8c489434dca8db70ebf7a1307a9: Deleting consensus metadata
I20260501 02:07:32.134548 32396 catalog_manager.cc:5002] TS 0b83e8c489434dca8db70ebf7a1307a9 (127.18.96.195:37809): tablet c149f7fe30f841a788b4c97c46b06555 (table test-table [id=127647a7fabd4830b91cbcc75ce6240b]) successfully deleted
I20260501 02:07:32.138060 479 log.cc:1199] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/c149f7fe30f841a788b4c97c46b06555
W20260501 02:07:32.138391 32396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:40431 (error 108)
I20260501 02:07:32.138567 479 ts_tablet_manager.cc:1950] T c149f7fe30f841a788b4c97c46b06555 P 766546e703394d269fa506098977e331: Deleting consensus metadata
I20260501 02:07:32.139771 32396 catalog_manager.cc:5002] TS 766546e703394d269fa506098977e331 (127.18.96.194:42699): tablet c149f7fe30f841a788b4c97c46b06555 (table test-table [id=127647a7fabd4830b91cbcc75ce6240b]) successfully deleted
I20260501 02:07:32.140115 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32621
I20260501 02:07:32.141075 449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:32.140648 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40431 (local address 127.0.0.1:34812)
0501 02:07:32.140916 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:32.140977 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:40431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
W20260501 02:07:32.141240 32396 catalog_manager.cc:4729] TS b14ec47c95eb4b519535b91a2e6e815f (127.18.96.193:40431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c149f7fe30f841a788b4c97c46b06555: Network error: Client connection negotiation failed: client connection to 127.18.96.193:40431: connect: Connection refused (error 111)
I20260501 02:07:32.148675 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32754
I20260501 02:07:32.157912 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 32376
I20260501 02:07:32.167425 18819 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260501 02:07:32.512439 18819 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260501 02:07:32.512579 18819 mini_ranger.cc:66] Stopping Ranger...
I20260501 02:07:32.857654 18819 mini_ranger.cc:68] Stopped Ranger
2026-05-01 02:07:32.857 UTC [31143] LOG: received smart shutdown request
2026-05-01 02:07:32.861 UTC [31143] LOG: background worker "logical replication launcher" (PID 31151) exited with exit code 1
2026-05-01 02:07:32.861 UTC [31146] LOG: shutting down
2026-05-01 02:07:32.863 UTC [31146] LOG: checkpoint starting: shutdown immediate
2026-05-01 02:07:33.814 UTC [31146] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.036 s, sync=0.908 s, total=0.953 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045108, redo lsn=0/2045108
2026-05-01 02:07:33.822 UTC [31143] LOG: database system is shut down
2026-05-01T02:07:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56234 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:34 dist-test-slave-2j90 krb5kdc[491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:34 dist-test-slave-2j90 krb5kdc[491](info): set up 2 sockets
May 01 02:07:34 dist-test-slave-2j90 krb5kdc[491](info): commencing operation
krb5kdc: starting...
W20260501 02:07:35.954473 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.989s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:07:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:07:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:36.090754 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45047
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:36.190920 507 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:36.191162 507 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:36.191264 507 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:36.194391 507 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:36.194463 507 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:36.194504 507 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:36.194566 507 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:36.194607 507 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:36.198750 507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:36.199698 507 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:36.200457 507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:36.205926 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:36.205991 507 server_base.cc:1061] running on GCE node
W20260501 02:07:36.206095 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:36.205991 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:36.206651 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:36.207453 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:36.208602 507 hybrid_clock.cc:648] HybridClock initialized: now 1777601256208579 us; error 37 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:36.211025 507 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:36.211992 507 webserver.cc:492] Webserver started at http://127.18.96.254:37871/ using document root <none> and password file <none>
I20260501 02:07:36.212450 507 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:36.212491 507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:36.212646 507 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:36.214243 507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "d95c03f8b33f4a45bd9f861807e51b5a"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "1765d3be3bb9bd396bfcecec8e6c7bbe"
server_key_iv: "bc22d2b158fb8514e11464df85948e27"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.214663 507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "d95c03f8b33f4a45bd9f861807e51b5a"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "1765d3be3bb9bd396bfcecec8e6c7bbe"
server_key_iv: "bc22d2b158fb8514e11464df85948e27"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.217613 507 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:07:36.219626 522 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:36.220485 507 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:36.220611 507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "d95c03f8b33f4a45bd9f861807e51b5a"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "1765d3be3bb9bd396bfcecec8e6c7bbe"
server_key_iv: "bc22d2b158fb8514e11464df85948e27"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.220701 507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:36.228744 507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:36.231178 507 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:36.231364 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:36.238489 507 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:45047
I20260501 02:07:36.238489 574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:45047 every 8 connection(s)
I20260501 02:07:36.239435 507 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:36.241797 575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:36.245486 575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.245783 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 507
I20260501 02:07:36.245901 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:07:36.246192 18819 external_mini_cluster.cc:1468] Setting key 3d4ff9941193971341d6c6c6a4465194
WARNING: no policy specified for kudu/127.18.96.253@KRBTEST.COM; defaulting to no policy
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.253@KRBTEST.COM" created.
I20260501 02:07:36.259415 579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.247581 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.18.96.254:45047)
0501 02:07:36.248110 (+ 529us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.248119 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.248268 (+ 149us) server_negotiation.cc:408] Connection header received
0501 02:07:36.249347 (+ 1079us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.249351 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.249493 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.249630 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.250792 (+ 1162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.251470 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.252310 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.252487 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.254911 (+ 2424us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.254928 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.254938 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.254959 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.256692 (+ 1733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.257227 (+ 535us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.257235 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.257237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.257317 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.257586 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.257592 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.257597 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.258014 (+ 417us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.258149 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.258469 (+ 320us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.258648 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":73600,"thread_start_us":167,"threads_started":1}
I20260501 02:07:36.259421 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.247581 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45047 (local address 127.0.0.1:48504)
0501 02:07:36.248105 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.248145 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.249062 (+ 917us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.249698 (+ 636us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.249703 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.250214 (+ 511us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.250639 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.250658 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.251599 (+ 941us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.251603 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.252175 (+ 572us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.252184 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.252408 (+ 224us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.253022 (+ 614us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.253043 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.254768 (+ 1725us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.256802 (+ 2034us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.256813 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.256824 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.257111 (+ 287us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.257425 (+ 314us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.257427 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.257428 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.257494 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.258122 (+ 628us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.258126 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.258226 (+ 100us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.258422 (+ 196us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.258714 (+ 292us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":95,"thread_start_us":154,"threads_started":1}
I20260501 02:07:36.261693 575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.262506 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.262253 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59428)
0501 02:07:36.262394 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.262449 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260501 02:07:36.262712 524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
I20260501 02:07:36.263478 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.263291 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59432)
0501 02:07:36.263393 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.263430 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260501 02:07:36.263700 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41913: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:36.294493 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41913
--webserver_interface=127.18.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:36.312901 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } attempt: 1
I20260501 02:07:36.313747 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.313467 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59436)
0501 02:07:36.313625 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.313676 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260501 02:07:36.314430 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.314225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59452)
0501 02:07:36.314333 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.314377 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260501 02:07:36.314555 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41913: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
I20260501 02:07:36.382745 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } attempt: 2
I20260501 02:07:36.383536 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.383255 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59462)
0501 02:07:36.383410 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.383464 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260501 02:07:36.384301 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.384057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59468)
0501 02:07:36.384190 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.384237 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260501 02:07:36.384434 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41913: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41913: connect: Connection refused (error 111)
W20260501 02:07:36.392143 584 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:36.392364 584 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:36.392472 584 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:36.396932 584 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:36.397007 584 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:36.397058 584 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:36.397096 584 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:36.397133 584 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:36.401253 584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:36.402318 584 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:36.403079 584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:36.408416 590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:36.408366 589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:36.408442 592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:36.408511 584 server_base.cc:1061] running on GCE node
I20260501 02:07:36.408998 584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:36.409838 584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:36.411019 584 hybrid_clock.cc:648] HybridClock initialized: now 1777601256410998 us; error 38 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:36.413512 584 init.cc:377] Logged in from keytab as kudu/127.18.96.253@KRBTEST.COM (short username kudu)
I20260501 02:07:36.414593 584 webserver.cc:492] Webserver started at http://127.18.96.253:39801/ using document root <none> and password file <none>
I20260501 02:07:36.415117 584 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:36.415191 584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:36.415407 584 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:36.417030 584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data/instance:
uuid: "fa17fe5e5e964bf8ae834a9490faab12"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "c75556257b418c6a845ca22657d31dec"
server_key_iv: "1a138644956106265aab8893f5ed321a"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.417506 584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal/instance:
uuid: "fa17fe5e5e964bf8ae834a9490faab12"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "c75556257b418c6a845ca22657d31dec"
server_key_iv: "1a138644956106265aab8893f5ed321a"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.420598 584 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260501 02:07:36.422608 599 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:36.423507 584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:36.423632 584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "fa17fe5e5e964bf8ae834a9490faab12"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "c75556257b418c6a845ca22657d31dec"
server_key_iv: "1a138644956106265aab8893f5ed321a"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.423727 584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:36.450217 584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:36.452693 584 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:36.452883 584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:36.460175 584 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.253:41913
I20260501 02:07:36.460183 651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.253:41913 every 8 connection(s)
I20260501 02:07:36.461120 584 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
I20260501 02:07:36.463605 652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:36.467121 652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.469883 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 584
I20260501 02:07:36.470038 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-1/wal/instance
I20260501 02:07:36.470301 18819 external_mini_cluster.cc:1468] Setting key ed7f7c0f516ba640ae76880c7df937c6
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:36.475590 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } attempt: 3
WARNING: no policy specified for kudu/127.18.96.252@KRBTEST.COM; defaulting to no policy
I20260501 02:07:36.480060 579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.468671 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.18.96.254:45047)
0501 02:07:36.468864 (+ 193us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.468869 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.469486 (+ 617us) server_negotiation.cc:408] Connection header received
0501 02:07:36.470463 (+ 977us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.470466 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.470512 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.470590 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.472104 (+ 1514us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.472841 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.473506 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.473762 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.476019 (+ 2257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.476041 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.476045 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.476074 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.477863 (+ 1789us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.478403 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.478407 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.478409 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.478459 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.478751 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.478754 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.478756 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.478928 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.479319 (+ 391us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.479753 (+ 434us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.479865 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260501 02:07:36.480559 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.468934 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45047 (local address 127.0.0.1:48506)
0501 02:07:36.469356 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.469387 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.470229 (+ 842us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.470770 (+ 541us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.470781 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.471350 (+ 569us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.471932 (+ 582us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.471950 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.472988 (+ 1038us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.472991 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.473383 (+ 392us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.473388 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.473542 (+ 154us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.474128 (+ 586us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.474146 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.475840 (+ 1694us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.478005 (+ 2165us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.478010 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.478021 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.478295 (+ 274us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.478558 (+ 263us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.478563 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.478569 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.478647 (+ 78us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.479312 (+ 665us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.479317 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.479516 (+ 199us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.479702 (+ 186us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.479894 (+ 192us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.252@KRBTEST.COM" created.
I20260501 02:07:36.481829 652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:07:36.487035 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.476008 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59470)
0501 02:07:36.476198 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.476212 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.476292 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.476708 (+ 416us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.476712 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.476888 (+ 176us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.477150 (+ 262us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.477160 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.478111 (+ 951us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.478115 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.478695 (+ 580us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.478704 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.478823 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.479267 (+ 444us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.479296 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.482523 (+ 3227us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.485060 (+ 2537us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.485064 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.485066 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.485235 (+ 169us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.485557 (+ 322us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.485564 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.485569 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.485636 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.486463 (+ 827us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.486468 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.486559 (+ 91us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.486683 (+ 124us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.486816 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260501 02:07:36.487104 656 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.476078 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.18.96.253:41913)
0501 02:07:36.476423 (+ 345us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.476427 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.476443 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:07:36.476474 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.476477 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.476584 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.476713 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.477291 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.477964 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.478814 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.479041 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.482670 (+ 3629us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.482692 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.482704 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.482728 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.484947 (+ 2219us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.485394 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.485399 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.485401 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.485448 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.485740 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.485744 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.485746 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.485954 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.486473 (+ 519us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.486779 (+ 306us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.486919 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:07:36.489300 575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.490151 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.489914 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38732)
0501 02:07:36.490047 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.490096 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260501 02:07:36.490856 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.490645 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38734)
0501 02:07:36.490769 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.490811 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260501 02:07:36.491039 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
I20260501 02:07:36.492224 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.482333 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59486)
0501 02:07:36.482478 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.482493 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.482579 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.484107 (+ 1528us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.484113 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.484278 (+ 165us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.484422 (+ 144us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.484427 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.485334 (+ 907us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.485338 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.485808 (+ 470us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.485815 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.485930 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.486422 (+ 492us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.486437 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.487743 (+ 1306us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.491165 (+ 3422us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.491169 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.491170 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.491328 (+ 158us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.491549 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.491551 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.491553 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.491591 (+ 38us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.491879 (+ 288us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.491881 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.491944 (+ 63us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.492021 (+ 77us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.492105 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260501 02:07:36.492304 657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.482546 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.18.96.253:41913)
0501 02:07:36.483843 (+ 1297us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.483849 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.483867 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:07:36.483905 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.483911 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.483966 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.484499 (+ 533us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.484567 (+ 68us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.485213 (+ 646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.485939 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.486239 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.487853 (+ 1614us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.487872 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.487877 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.487907 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.491017 (+ 3110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.491418 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.491420 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.491422 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.491461 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.491660 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.491662 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.491663 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.491812 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.491882 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.492071 (+ 189us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.492152 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":135,"threads_started":1}
I20260501 02:07:36.493005 652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.493609 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.493392 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38748)
0501 02:07:36.493501 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.493563 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260501 02:07:36.493798 602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
I20260501 02:07:36.494611 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.494418 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38756)
0501 02:07:36.494535 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.494569 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260501 02:07:36.494752 652 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:36.521677 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:35939
--webserver_interface=127.18.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:36.522184 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 1
I20260501 02:07:36.523114 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.522720 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38772)
0501 02:07:36.522932 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.523016 (+ 84us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260501 02:07:36.523885 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.523670 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38774)
0501 02:07:36.523789 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.523832 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260501 02:07:36.524160 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
I20260501 02:07:36.543974 652 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 1
I20260501 02:07:36.544798 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.544479 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38788)
0501 02:07:36.544650 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.544713 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260501 02:07:36.545495 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.545270 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38792)
0501 02:07:36.545391 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.545433 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260501 02:07:36.545631 652 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
I20260501 02:07:36.599337 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 2
I20260501 02:07:36.600158 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.599859 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38802)
0501 02:07:36.600023 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.600081 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260501 02:07:36.600843 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.600641 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38806)
0501 02:07:36.600748 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.600792 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260501 02:07:36.600962 575 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
I20260501 02:07:36.613790 652 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 2
I20260501 02:07:36.614550 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.614220 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38812)
0501 02:07:36.614404 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.614457 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260501 02:07:36.615231 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.615015 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38818)
0501 02:07:36.615126 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.615171 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260501 02:07:36.615370 652 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:35939: Network error: Client connection negotiation failed: client connection to 127.18.96.252:35939: connect: Connection refused (error 111)
W20260501 02:07:36.620965 662 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:36.621125 662 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:36.621171 662 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:36.624454 662 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:36.624502 662 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:36.624555 662 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:36.624593 662 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:36.624647 662 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:36.628876 662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45047,127.18.96.253:41913,127.18.96.252:35939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:35939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:36.629801 662 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:36.630606 662 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:36.635763 667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:36.635767 670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:36.635767 668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:36.636091 662 server_base.cc:1061] running on GCE node
I20260501 02:07:36.636547 662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:36.637312 662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:36.638487 662 hybrid_clock.cc:648] HybridClock initialized: now 1777601256638476 us; error 33 us; skew 500 ppm
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:36.640869 662 init.cc:377] Logged in from keytab as kudu/127.18.96.252@KRBTEST.COM (short username kudu)
I20260501 02:07:36.641794 662 webserver.cc:492] Webserver started at http://127.18.96.252:39549/ using document root <none> and password file <none>
I20260501 02:07:36.642364 662 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:36.642436 662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:36.642643 662 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:36.644225 662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data/instance:
uuid: "9ea2b150b2334739bd1074a8be9bf93b"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "582081a8ddc5a08257f725712954267d"
server_key_iv: "e035b1a73e59bed57a0b656b8244ea71"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.644688 662 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal/instance:
uuid: "9ea2b150b2334739bd1074a8be9bf93b"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "582081a8ddc5a08257f725712954267d"
server_key_iv: "e035b1a73e59bed57a0b656b8244ea71"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.647692 662 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:07:36.649535 677 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:36.650478 662 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:36.650602 662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "9ea2b150b2334739bd1074a8be9bf93b"
format_stamp: "Formatted at 2026-05-01 02:07:36 on dist-test-slave-2j90"
server_key: "582081a8ddc5a08257f725712954267d"
server_key_iv: "e035b1a73e59bed57a0b656b8244ea71"
server_key_version: "encryptionkey@0"
I20260501 02:07:36.650694 662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:36.662726 662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:36.664893 662 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:36.665071 662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:36.671816 662 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.252:35939
I20260501 02:07:36.671833 729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.252:35939 every 8 connection(s)
I20260501 02:07:36.672689 662 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
I20260501 02:07:36.675154 730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:36.676869 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 662
I20260501 02:07:36.677016 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACert.1777600936179445-18819-0/minicluster-data/master-2/wal/instance
I20260501 02:07:36.677309 18819 external_mini_cluster.cc:1468] Setting key 720aab82f7ef8aa87ddd0f5b037e0c57
I20260501 02:07:36.679271 730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:36.689380 579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.681071 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.18.96.254:45047)
0501 02:07:36.681199 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.681204 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.681448 (+ 244us) server_negotiation.cc:408] Connection header received
0501 02:07:36.681612 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.681615 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.681662 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.681748 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.682583 (+ 835us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.683093 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.683923 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.684057 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.686086 (+ 2029us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.686110 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.686112 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.686135 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.687570 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.688115 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.688118 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.688120 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.688165 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.688451 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.688453 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.688475 (+ 22us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.688629 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.688740 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.689121 (+ 381us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.689231 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:07:36.690021 733 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.678510 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.18.96.252:35939)
0501 02:07:36.678900 (+ 390us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.678909 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.678947 (+ 38us) server_negotiation.cc:408] Connection header received
0501 02:07:36.679573 (+ 626us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.679594 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.679905 (+ 311us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.680199 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.681114 (+ 915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.682080 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.682716 (+ 636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.682989 (+ 273us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.685469 (+ 2480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.685484 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.685491 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.685512 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.687437 (+ 1925us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.687913 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.687920 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.687922 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.687988 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.688256 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.688259 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.688261 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.688626 (+ 365us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.688800 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.689009 (+ 209us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.689225 (+ 216us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":3712,"thread_start_us":106,"threads_started":1}
I20260501 02:07:36.690019 734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.681001 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45047 (local address 127.0.0.1:48522)
0501 02:07:36.681343 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.681370 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.681496 (+ 126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.681799 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.681806 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.682214 (+ 408us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.682445 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.682450 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.683209 (+ 759us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.683212 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.683782 (+ 570us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.683791 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.683979 (+ 188us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.684456 (+ 477us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.684473 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.685909 (+ 1436us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.687682 (+ 1773us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.687687 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.687692 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.688009 (+ 317us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.688256 (+ 247us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.688259 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.688261 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.688359 (+ 98us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.688739 (+ 380us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.688743 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.688862 (+ 119us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.689042 (+ 180us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.689283 (+ 241us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":108,"spinlock_wait_cycles":32512,"thread_start_us":101,"threads_started":1}
I20260501 02:07:36.691643 730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:36.699469 734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.692486 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41913 (local address 127.0.0.1:59490)
0501 02:07:36.692906 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.692917 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.692985 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.693284 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.693286 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.693414 (+ 128us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.693573 (+ 159us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.693578 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.694347 (+ 769us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.694352 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.694682 (+ 330us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.694688 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.694815 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.695304 (+ 489us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.695315 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.696504 (+ 1189us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.698213 (+ 1709us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.698218 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.698220 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.698387 (+ 167us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.698631 (+ 244us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.698633 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.698635 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.698674 (+ 39us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.699030 (+ 356us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.699033 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.699110 (+ 77us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.699221 (+ 111us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.699327 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357}
I20260501 02:07:36.699643 657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.692342 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.18.96.253:41913)
0501 02:07:36.692549 (+ 207us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.692553 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.692965 (+ 412us) server_negotiation.cc:408] Connection header received
0501 02:07:36.693090 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.693096 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.693178 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.693262 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.693697 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.694230 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.694803 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.695017 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.696627 (+ 1610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.696647 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.696650 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.696678 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.698127 (+ 1449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.698486 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.698490 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.698492 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.698539 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.698761 (+ 222us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.698765 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.698768 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.698931 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.699022 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.699295 (+ 273us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.699423 (+ 128us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260501 02:07:36.700111 575 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 3
I20260501 02:07:36.700312 730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:07:36.701175 579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.692076 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.18.96.254:45047)
0501 02:07:36.692185 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.692188 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.692253 (+ 65us) server_negotiation.cc:408] Connection header received
0501 02:07:36.692460 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.692464 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.692503 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.692555 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.693571 (+ 1016us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.694042 (+ 471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.694628 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.694835 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.697253 (+ 2418us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.697267 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.697269 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.697307 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.698919 (+ 1612us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.699352 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.699355 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.699357 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.699402 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.699834 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.699837 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.699838 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.699998 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.700790 (+ 792us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.700864 (+ 74us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.701003 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:07:36.706542 652 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } attempt: 3
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:07:36.709075 733 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.700807 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.18.96.252:35939)
0501 02:07:36.700932 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.700936 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.701253 (+ 317us) server_negotiation.cc:408] Connection header received
0501 02:07:36.701368 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.701371 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.701403 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.701447 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.702016 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.702789 (+ 773us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.703323 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.703527 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.705367 (+ 1840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.705388 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.705392 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.705418 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.707395 (+ 1977us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.707882 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.707885 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.707887 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.707926 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.708160 (+ 234us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.708163 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.708165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.708312 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.708374 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.708755 (+ 381us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.708868 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260501 02:07:36.709635 734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.700738 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38842)
0501 02:07:36.701169 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.701182 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.701276 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.701554 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.701556 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.701719 (+ 163us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.701890 (+ 171us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.701899 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.702898 (+ 999us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.702901 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.703207 (+ 306us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.703212 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.703338 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.703826 (+ 488us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.703838 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.705237 (+ 1399us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.707527 (+ 2290us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.707532 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.707536 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.707781 (+ 245us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.708023 (+ 242us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.708028 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.708031 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.708074 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.708431 (+ 357us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.708433 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.708538 (+ 105us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.709339 (+ 801us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.709454 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260501 02:07:36.710433 578 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.700490 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38836)
0501 02:07:36.700831 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.700843 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.700940 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.701353 (+ 413us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.701359 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.701543 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.701725 (+ 182us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.701730 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.703576 (+ 1846us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.703581 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.704083 (+ 502us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.704102 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.704166 (+ 64us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.704581 (+ 415us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.704596 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.706005 (+ 1409us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.708952 (+ 2947us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.708955 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.708958 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.709193 (+ 235us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.709451 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.709454 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.709456 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.709505 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.710064 (+ 559us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.710067 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.710147 (+ 80us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.710227 (+ 80us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.710331 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265}
I20260501 02:07:36.710925 735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.700867 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.18.96.252:35939)
0501 02:07:36.701141 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.701144 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.701153 (+ 9us) server_negotiation.cc:408] Connection header received
0501 02:07:36.701178 (+ 25us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.701180 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.701209 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.701325 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.701833 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.703437 (+ 1604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.704382 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.704599 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.706160 (+ 1561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.706177 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.706181 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.706207 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.708817 (+ 2610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.709302 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.709305 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.709307 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.709349 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.709619 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.709622 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.709624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.709802 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.710623 (+ 821us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.710695 (+ 72us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.710771 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":113,"threads_started":1}
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:07:36.712774 657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.702948 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.18.96.253:41913)
0501 02:07:36.703045 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.703049 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.703068 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:07:36.703094 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.703096 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.703127 (+ 31us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.703215 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.704080 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.705018 (+ 938us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.705661 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.705857 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.708310 (+ 2453us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.708326 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.708329 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.708355 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.711273 (+ 2918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.711713 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.711717 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.711719 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.711765 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.712011 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.712014 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.712016 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.712170 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.712345 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.712532 (+ 187us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.712611 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260501 02:07:36.713670 730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: Bootstrap starting.
I20260501 02:07:36.716099 654 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.706920 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:35939 (local address 127.0.0.1:38852)
0501 02:07:36.707085 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:36.707101 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:36.707178 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:36.707586 (+ 408us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:36.707592 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:36.707764 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:36.707980 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.707989 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.709087 (+ 1098us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.709094 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:36.709616 (+ 522us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:36.709624 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.709696 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.710108 (+ 412us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:36.710128 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:36.712170 (+ 2042us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:36.714670 (+ 2500us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.714674 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.714676 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.714913 (+ 237us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.715195 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:36.715198 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:36.715200 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:36.715267 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:36.715652 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:36.715656 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:36.715735 (+ 79us) client_negotiation.cc:770] Sending connection context
0501 02:07:36.715844 (+ 109us) client_negotiation.cc:241] Negotiation successful
0501 02:07:36.715964 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260501 02:07:36.716225 736 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.706968 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.18.96.252:35939)
0501 02:07:36.707342 (+ 374us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.707346 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.707362 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:07:36.707395 (+ 33us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.707398 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.707451 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.708098 (+ 647us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.708199 (+ 101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.708954 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.709892 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.710145 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.712754 (+ 2609us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.712769 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.712772 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.712795 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.714564 (+ 1769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.715027 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.715032 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.715035 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.715086 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.715381 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.715384 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.715386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.715545 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.715643 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.715914 (+ 271us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.716021 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":97,"threads_started":1}
I20260501 02:07:36.716254 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:07:36.716562 730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:36.717283 730 log.cc:826] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:36.719552 730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: No bootstrap required, opened a new log
I20260501 02:07:36.720238 575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: Bootstrap starting.
I20260501 02:07:36.720245 652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: Bootstrap starting.
I20260501 02:07:36.722275 652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:36.722406 575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:36.722590 730 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.722847 730 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:36.722949 730 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea2b150b2334739bd1074a8be9bf93b, State: Initialized, Role: FOLLOWER
I20260501 02:07:36.723129 652 log.cc:826] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:36.723205 575 log.cc:826] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:36.723466 730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.724803 730 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:36.725239 575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: No bootstrap required, opened a new log
I20260501 02:07:36.725248 652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: No bootstrap required, opened a new log
I20260501 02:07:36.727867 741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.728029 741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.729279 575 raft_consensus.cc:359] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.729504 575 raft_consensus.cc:385] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:36.729630 575 raft_consensus.cc:740] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d95c03f8b33f4a45bd9f861807e51b5a, State: Initialized, Role: FOLLOWER
W20260501 02:07:36.730010 753 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:07:36.730118 753 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:07:36.730096 575 consensus_queue.cc:260] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.731263 575 sys_catalog.cc:565] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:36.731217 652 raft_consensus.cc:359] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.731424 652 raft_consensus.cc:385] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:36.731520 652 raft_consensus.cc:740] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa17fe5e5e964bf8ae834a9490faab12, State: Initialized, Role: FOLLOWER
I20260501 02:07:36.729000 662 master_runner.cc:186] Error getting master registration for 127.18.96.254:45047: OK, instance_id { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" instance_seqno: 1777601256236492 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 02:07:36.731753 662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:07:36.731940 652 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
W20260501 02:07:36.732560 528 tablet.cc:2404] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:07:36.733250 652 sys_catalog.cc:565] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: configured and running, proceeding with master startup.
W20260501 02:07:36.736367 767 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:07:36.736519 767 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:07:36.738188 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.738298 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: This master's current role is: FOLLOWER
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260501 02:07:36.738746 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.738837 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.742502 584 master_runner.cc:263] No leader master found from master fa17fe5e5e964bf8ae834a9490faab12
I20260501 02:07:36.742568 584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:07:36.743290 507 master_runner.cc:263] No leader master found from master d95c03f8b33f4a45bd9f861807e51b5a
I20260501 02:07:36.743343 507 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260501 02:07:36.748555 779 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:07:36.748627 779 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:07:36.758438 736 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.748195 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.18.96.252:35939)
0501 02:07:36.748311 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.748315 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.748872 (+ 557us) server_negotiation.cc:408] Connection header received
0501 02:07:36.749045 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.749048 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.749093 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.749143 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.750253 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.750982 (+ 729us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.751845 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.752044 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.754994 (+ 2950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.755012 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.755015 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.755040 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.756756 (+ 1716us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.757212 (+ 456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.757228 (+ 16us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.757230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.757273 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.757509 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.757513 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.757515 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.757681 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.757796 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.758150 (+ 354us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.758259 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:07:36.768294 657 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.748266 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.18.96.253:41913)
0501 02:07:36.748375 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.748379 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.759117 (+ 10738us) server_negotiation.cc:408] Connection header received
0501 02:07:36.759272 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.759275 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.759329 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.759381 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.760295 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.760966 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.761749 (+ 783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.761986 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.764719 (+ 2733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.764739 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.764742 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.764771 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.766456 (+ 1685us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.766954 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.766960 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.766963 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.767032 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.767316 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.767324 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.767328 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.767569 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.767677 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.768017 (+ 340us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.768121 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260501 02:07:36.769775 579 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:36.747967 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.18.96.254:45047)
0501 02:07:36.748084 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:36.748089 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:36.758191 (+ 10102us) server_negotiation.cc:408] Connection header received
0501 02:07:36.758382 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:36.758385 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:36.758435 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:36.758489 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:36.759395 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.760099 (+ 704us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:36.760926 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:36.761119 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:36.766358 (+ 5239us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:36.766381 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:36.766386 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:36.766421 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:36.768098 (+ 1677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.768538 (+ 440us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.768542 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.768543 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.768620 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:36.768877 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:36.768881 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:36.768883 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:36.769060 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:36.769150 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:36.769495 (+ 345us) server_negotiation.cc:300] Negotiation successful
0501 02:07:36.769603 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260501 02:07:36.809010 741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:07:36.809212 741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.810281 741 leader_election.cc:290] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d95c03f8b33f4a45bd9f861807e51b5a (127.18.96.254:45047), fa17fe5e5e964bf8ae834a9490faab12 (127.18.96.253:41913)
I20260501 02:07:36.811213 549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea2b150b2334739bd1074a8be9bf93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" is_pre_election: true
I20260501 02:07:36.811462 549 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea2b150b2334739bd1074a8be9bf93b in term 0.
I20260501 02:07:36.811821 626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea2b150b2334739bd1074a8be9bf93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa17fe5e5e964bf8ae834a9490faab12" is_pre_election: true
I20260501 02:07:36.812022 626 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea2b150b2334739bd1074a8be9bf93b in term 0.
I20260501 02:07:36.812588 679 leader_election.cc:304] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ea2b150b2334739bd1074a8be9bf93b, fa17fe5e5e964bf8ae834a9490faab12; no voters:
I20260501 02:07:36.813205 741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:07:36.813272 741 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:07:36.813315 741 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:36.814743 741 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.815073 741 leader_election.cc:290] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [CANDIDATE]: Term 1 election: Requested vote from peers d95c03f8b33f4a45bd9f861807e51b5a (127.18.96.254:45047), fa17fe5e5e964bf8ae834a9490faab12 (127.18.96.253:41913)
I20260501 02:07:36.815724 626 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea2b150b2334739bd1074a8be9bf93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa17fe5e5e964bf8ae834a9490faab12"
I20260501 02:07:36.815833 626 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:36.817018 626 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea2b150b2334739bd1074a8be9bf93b in term 1.
I20260501 02:07:36.817358 679 leader_election.cc:304] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ea2b150b2334739bd1074a8be9bf93b, fa17fe5e5e964bf8ae834a9490faab12; no voters:
I20260501 02:07:36.817607 549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ea2b150b2334739bd1074a8be9bf93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d95c03f8b33f4a45bd9f861807e51b5a"
I20260501 02:07:36.817729 549 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:36.818946 549 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea2b150b2334739bd1074a8be9bf93b in term 1.
I20260501 02:07:36.821507 741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:36.821784 741 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [term 1 LEADER]: Becoming Leader. State: Replica: 9ea2b150b2334739bd1074a8be9bf93b, State: Running, Role: LEADER
I20260501 02:07:36.822182 741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } }
I20260501 02:07:36.823952 741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea2b150b2334739bd1074a8be9bf93b. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.824139 741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:36.824844 790 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:36.825909 790 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:36.832988 626 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [term 1 FOLLOWER]: Refusing update from remote peer 9ea2b150b2334739bd1074a8be9bf93b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:36.836326 788 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:36.840891 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea2b150b2334739bd1074a8be9bf93b. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.841005 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.842018 549 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [term 1 FOLLOWER]: Refusing update from remote peer 9ea2b150b2334739bd1074a8be9bf93b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:36.842576 788 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:36.846203 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea2b150b2334739bd1074a8be9bf93b. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.846316 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.851871 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.851962 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:36.852984 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.853086 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.853459 791 mvcc.cc:204] Tried to move back new op lower bound from 7281054747980656640 to 7281054747947188224. Current Snapshot: MvccSnapshot[applied={T|T < 7281054747980656640}]
I20260501 02:07:36.856009 794 mvcc.cc:204] Tried to move back new op lower bound from 7281054747980656640 to 7281054747947188224. Current Snapshot: MvccSnapshot[applied={T|T < 7281054747980656640}]
I20260501 02:07:36.856734 741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.856827 741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:36.857697 759 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ea2b150b2334739bd1074a8be9bf93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d95c03f8b33f4a45bd9f861807e51b5a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45047 } } peers { permanent_uuid: "fa17fe5e5e964bf8ae834a9490faab12" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41913 } } peers { permanent_uuid: "9ea2b150b2334739bd1074a8be9bf93b" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 35939 } } }
I20260501 02:07:36.857790 759 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:07:36.863128 792 mvcc.cc:204] Tried to move back new op lower bound from 7281054747980656640 to 7281054747947188224. Current Snapshot: MvccSnapshot[applied={T|T < 7281054747980656640}]
I20260501 02:07:36.863415 790 catalog_manager.cc:1357] Generated new cluster ID: a79d1696ae1c45379ce7dc58118b8bf3
I20260501 02:07:36.863466 790 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:07:36.895004 790 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:07:36.896133 790 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:36.912783 790 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9ea2b150b2334739bd1074a8be9bf93b: Generated new TSK 0
I20260501 02:07:36.913496 790 catalog_manager.cc:1524] Initializing in-progress tserver states...
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.18.96.252@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.18.96.254@KRBTEST.COM
May 01 02:07:36 dist-test-slave-2j90 krb5kdc[491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.18.96.253@KRBTEST.COM
I20260501 02:07:37.737715 767 catalog_manager.cc:1269] Loaded cluster ID: a79d1696ae1c45379ce7dc58118b8bf3
I20260501 02:07:37.737794 767 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: loading cluster ID for follower catalog manager: success
I20260501 02:07:37.739405 767 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: acquiring CA information for follower catalog manager: success
I20260501 02:07:37.740394 767 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d95c03f8b33f4a45bd9f861807e51b5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:07:37.749583 779 catalog_manager.cc:1269] Loaded cluster ID: a79d1696ae1c45379ce7dc58118b8bf3
I20260501 02:07:37.749634 779 catalog_manager.cc:1562] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: loading cluster ID for follower catalog manager: success
I20260501 02:07:37.751055 779 catalog_manager.cc:1584] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: acquiring CA information for follower catalog manager: success
I20260501 02:07:37.751933 779 catalog_manager.cc:1612] T 00000000000000000000000000000000 P fa17fe5e5e964bf8ae834a9490faab12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:07:38.041589 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 507
I20260501 02:07:38.048225 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 584
I20260501 02:07:38.054117 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 662
2026-05-01T02:07:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4110 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:38 dist-test-slave-2j90 krb5kdc[805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:38 dist-test-slave-2j90 krb5kdc[805](info): set up 2 sockets
May 01 02:07:38 dist-test-slave-2j90 krb5kdc[805](info): commencing operation
krb5kdc: starting...
W20260501 02:07:40.034705 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.959s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:07:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:40.166671 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45117
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:40.263129 821 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:40.263399 821 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:40.263481 821 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:40.266782 821 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:40.266877 821 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:40.266919 821 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:40.266978 821 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:40.267020 821 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:40.271176 821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:40.272254 821 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:40.273068 821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:40.278375 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:40.278420 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:40.278375 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:40.278826 821 server_base.cc:1061] running on GCE node
I20260501 02:07:40.279254 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:40.280205 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:40.281383 821 hybrid_clock.cc:648] HybridClock initialized: now 1777601260281363 us; error 40 us; skew 500 ppm
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601260, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:40.284107 821 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:40.285176 821 webserver.cc:492] Webserver started at http://127.18.96.254:34647/ using document root <none> and password file <none>
I20260501 02:07:40.285719 821 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:40.285791 821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:40.286042 821 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:40.287645 821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "1c222477d000493280276f8ba502fe70"
format_stamp: "Formatted at 2026-05-01 02:07:40 on dist-test-slave-2j90"
server_key: "c447bb95d85b3c6ed94cc630baaf8a9e"
server_key_iv: "e63c0b982a4225300e0b43f31eec0225"
server_key_version: "encryptionkey@0"
I20260501 02:07:40.288139 821 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "1c222477d000493280276f8ba502fe70"
format_stamp: "Formatted at 2026-05-01 02:07:40 on dist-test-slave-2j90"
server_key: "c447bb95d85b3c6ed94cc630baaf8a9e"
server_key_iv: "e63c0b982a4225300e0b43f31eec0225"
server_key_version: "encryptionkey@0"
I20260501 02:07:40.291357 821 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:07:40.293365 836 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:40.294320 821 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:07:40.294445 821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "1c222477d000493280276f8ba502fe70"
format_stamp: "Formatted at 2026-05-01 02:07:40 on dist-test-slave-2j90"
server_key: "c447bb95d85b3c6ed94cc630baaf8a9e"
server_key_iv: "e63c0b982a4225300e0b43f31eec0225"
server_key_version: "encryptionkey@0"
I20260501 02:07:40.294548 821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:40.309679 821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:40.315994 821 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:40.316186 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:40.323233 821 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:45117
I20260501 02:07:40.323244 888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:45117 every 8 connection(s)
I20260501 02:07:40.324149 821 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:40.326786 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:40.331476 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: Bootstrap starting.
I20260501 02:07:40.331552 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 821
I20260501 02:07:40.331688 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:07:40.331919 18819 external_mini_cluster.cc:1468] Setting key ee6d91bff2711644f366ec1a9085a0b4
I20260501 02:07:40.333901 889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:40.334761 889 log.cc:826] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:40.336709 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: No bootstrap required, opened a new log
I20260501 02:07:40.339354 889 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } }
I20260501 02:07:40.339566 889 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:40.339635 889 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c222477d000493280276f8ba502fe70, State: Initialized, Role: FOLLOWER
I20260501 02:07:40.340029 889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } }
I20260501 02:07:40.340180 889 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:40.340240 889 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:40.340327 889 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:40.341413 889 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } }
I20260501 02:07:40.341753 889 leader_election.cc:304] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c222477d000493280276f8ba502fe70; no voters:
I20260501 02:07:40.342059 889 leader_election.cc:290] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:07:40.342387 894 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:40.342648 894 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [term 1 LEADER]: Becoming Leader. State: Replica: 1c222477d000493280276f8ba502fe70, State: Running, Role: LEADER
I20260501 02:07:40.342929 894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } }
I20260501 02:07:40.343106 889 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:40.344077 894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c222477d000493280276f8ba502fe70. Latest consensus state: current_term: 1 leader_uuid: "1c222477d000493280276f8ba502fe70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } } }
I20260501 02:07:40.344215 894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:40.344564 895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c222477d000493280276f8ba502fe70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c222477d000493280276f8ba502fe70" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45117 } } }
I20260501 02:07:40.344657 895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:40.345254 892 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:40.333009 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.18.96.254:45117)
0501 02:07:40.333397 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:40.333407 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:40.333442 (+ 35us) server_negotiation.cc:408] Connection header received
0501 02:07:40.334150 (+ 708us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:40.334178 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:40.334515 (+ 337us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:40.334827 (+ 312us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:40.335869 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:40.337003 (+ 1134us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:40.337667 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:40.337985 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:40.340182 (+ 2197us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:40.340205 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:40.340219 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:40.340253 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:40.342165 (+ 1912us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:40.342694 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:40.342702 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:40.342709 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:40.342793 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:40.343071 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:40.343075 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:40.343076 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:40.343479 (+ 403us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:40.343653 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:40.344147 (+ 494us) server_negotiation.cc:300] Negotiation successful
0501 02:07:40.344398 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260501 02:07:40.347911 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260501 02:07:40.347903 909 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:07:40.347980 909 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:07:40.348034 903 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:40.348869 903 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:40.357616 903 catalog_manager.cc:1357] Generated new cluster ID: 6693dbb4c4544a708ad1f7364df06391
I20260501 02:07:40.357681 903 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260501 02:07:40.367668 903 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:07:40.368624 903 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:40.373363 903 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1c222477d000493280276f8ba502fe70: Generated new TSK 0
I20260501 02:07:40.373907 903 catalog_manager.cc:1524] Initializing in-progress tserver states...
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.18.96.254@KRBTEST.COM
I20260501 02:07:40.428243 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 821
2026-05-01T02:07:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2372 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[918](info): set up 2 sockets
May 01 02:07:40 dist-test-slave-2j90 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260501 02:07:42.390458 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.942s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:07:42 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:07:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:42.526239 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:42.622323 934 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:42.622565 934 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:42.622661 934 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:42.625906 934 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:42.626010 934 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:42.626065 934 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:42.626130 934 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:42.626173 934 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:42.630214 934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:42.631156 934 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:42.631916 934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:42.637050 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:42.637082 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:42.637050 939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:42.637589 934 server_base.cc:1061] running on GCE node
I20260501 02:07:42.638024 934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:42.638772 934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:42.639921 934 hybrid_clock.cc:648] HybridClock initialized: now 1777601262639899 us; error 35 us; skew 500 ppm
May 01 02:07:42 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601262, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:42.642445 934 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:42.643424 934 webserver.cc:492] Webserver started at http://127.18.96.254:43137/ using document root <none> and password file <none>
I20260501 02:07:42.643977 934 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:42.644025 934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:42.644228 934 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:42.645789 934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "55d1676472604eb8bcddebac42c42215"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "e74ed1f6ecbce5080f995726ab2f2d79"
server_key_iv: "feecc70b5c443617954cc0831bcaef1f"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.646298 934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "55d1676472604eb8bcddebac42c42215"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "e74ed1f6ecbce5080f995726ab2f2d79"
server_key_iv: "feecc70b5c443617954cc0831bcaef1f"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.649240 934 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:07:42.651156 949 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:42.652011 934 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:42.652141 934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "55d1676472604eb8bcddebac42c42215"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "e74ed1f6ecbce5080f995726ab2f2d79"
server_key_iv: "feecc70b5c443617954cc0831bcaef1f"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.652225 934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:42.662449 934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:42.664567 934 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:42.664743 934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:42.671226 934 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40969
I20260501 02:07:42.671226 1001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40969 every 8 connection(s)
I20260501 02:07:42.672081 934 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:42.674575 1002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:42.679323 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap starting.
I20260501 02:07:42.680992 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 934
I20260501 02:07:42.681129 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:07:42.681394 18819 external_mini_cluster.cc:1468] Setting key cd64fbdcc696cf2225b37d0c81050753
I20260501 02:07:42.681476 1002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:42.682298 1002 log.cc:826] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:42.684129 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: No bootstrap required, opened a new log
I20260501 02:07:42.686970 1002 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:42.687222 1002 raft_consensus.cc:385] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:42.687280 1002 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Initialized, Role: FOLLOWER
I20260501 02:07:42.687783 1002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:42.687927 1002 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:42.687992 1002 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:42.688086 1002 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 0 FOLLOWER]: Advancing to term 1
May 01 02:07:42 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:42.689153 1002 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:42.689539 1002 leader_election.cc:304] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d1676472604eb8bcddebac42c42215; no voters:
I20260501 02:07:42.689810 1002 leader_election.cc:290] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:07:42.689978 1007 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:42.690200 1007 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 LEADER]: Becoming Leader. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Running, Role: LEADER
I20260501 02:07:42.690502 1007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:42.691010 1002 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:42.691541 1009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d1676472604eb8bcddebac42c42215. Latest consensus state: current_term: 1 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:42.691666 1009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:42.691855 1008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:42.691949 1008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:42.692196 1012 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:42.695112 1012 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:42.695194 1005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:42.682403 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.18.96.254:40969)
0501 02:07:42.682788 (+ 385us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:42.682796 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:42.682837 (+ 41us) server_negotiation.cc:408] Connection header received
0501 02:07:42.683330 (+ 493us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:42.683343 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:42.683573 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.683798 (+ 225us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:42.684929 (+ 1131us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.685657 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.686560 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.686756 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.689442 (+ 2686us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:42.689458 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:42.689467 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:42.689493 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:42.692335 (+ 2842us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.692817 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.692826 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:42.692834 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:42.692943 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.693223 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.693228 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:42.693232 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:42.693664 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:42.693833 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:42.694305 (+ 472us) server_negotiation.cc:300] Negotiation successful
0501 02:07:42.694571 (+ 266us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260501 02:07:42.700345 1012 catalog_manager.cc:1357] Generated new cluster ID: 0108d21bfea14acc82b6063388e2021f
I20260501 02:07:42.700407 1012 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:07:42.718470 1012 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:07:42.719195 1012 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:42.725497 1012 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Generated new TSK 0
I20260501 02:07:42.726014 1012 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:42.779084 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:42.877579 1030 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:42.877770 1030 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:42.877861 1030 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:42.881068 1030 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:42.881124 1030 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:42.881265 1030 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:07:42.885908 1030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:42.886897 1030 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:42.887722 1030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:42.893796 1036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:42.893836 1038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:42.893811 1035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:42.894166 1030 server_base.cc:1061] running on GCE node
I20260501 02:07:42.894492 1030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:42.894974 1030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:42.896143 1030 hybrid_clock.cc:648] HybridClock initialized: now 1777601262896131 us; error 36 us; skew 500 ppm
May 01 02:07:42 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601262, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:42.898685 1030 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:07:42.899619 1030 webserver.cc:492] Webserver started at http://127.18.96.193:34277/ using document root <none> and password file <none>
I20260501 02:07:42.900074 1030 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:42.900117 1030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:42.900251 1030 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:42.901705 1030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "77ec954487e04de09106f2bef8dda8e4"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "69cdfd91847681dc6c20a14dd682d2f7"
server_key_iv: "3164baff08ad191c1f3fbde45798233e"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.902145 1030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "77ec954487e04de09106f2bef8dda8e4"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "69cdfd91847681dc6c20a14dd682d2f7"
server_key_iv: "3164baff08ad191c1f3fbde45798233e"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.905161 1030 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260501 02:07:42.907049 1045 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:42.907907 1030 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:42.907985 1030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "77ec954487e04de09106f2bef8dda8e4"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "69cdfd91847681dc6c20a14dd682d2f7"
server_key_iv: "3164baff08ad191c1f3fbde45798233e"
server_key_version: "encryptionkey@0"
I20260501 02:07:42.908048 1030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:42.929937 1030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:42.932139 1030 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:42.932331 1030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:42.932822 1030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:42.933621 1030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:42.933684 1030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:42.933748 1030 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:42.933779 1030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:42.942579 1030 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:39369
I20260501 02:07:42.942605 1158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:39369 every 8 connection(s)
I20260501 02:07:42.943459 1030 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:07:42.944140 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1030
I20260501 02:07:42.944257 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:07:42.944546 18819 external_mini_cluster.cc:1468] Setting key 43e7d7bbae5cabf6460a8b67fca8f8dd
May 01 02:07:42 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601262, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:07:42.956230 1005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:42.945214 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:57761 (local address 127.18.96.254:40969)
0501 02:07:42.945374 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:42.945378 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:42.946067 (+ 689us) server_negotiation.cc:408] Connection header received
0501 02:07:42.946864 (+ 797us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:42.946871 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:42.946931 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:42.947024 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:42.948078 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.948609 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.949579 (+ 970us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.949724 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.952831 (+ 3107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:42.952854 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:42.952857 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:42.952879 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:42.954316 (+ 1437us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.954815 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.954819 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:42.954821 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:42.954868 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:42.955134 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:42.955138 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:42.955140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:42.955308 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:42.955410 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:42.955940 (+ 530us) server_negotiation.cc:300] Negotiation successful
0501 02:07:42.956065 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:07:42.956837 1161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:42.945507 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:57761)
0501 02:07:42.945918 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:42.945972 (+ 54us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:42.946686 (+ 714us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:42.947151 (+ 465us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:42.947158 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:42.947514 (+ 356us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:42.947940 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:42.947949 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.948735 (+ 786us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:42.948742 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:42.949425 (+ 683us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:42.949454 (+ 29us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:42.949696 (+ 242us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:42.950901 (+ 1205us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:42.950929 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:42.952675 (+ 1746us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:42.954437 (+ 1762us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.954443 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:42.954452 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:42.954715 (+ 263us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.954957 (+ 242us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:42.954960 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:42.954962 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:42.955045 (+ 83us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:42.955461 (+ 416us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:42.955465 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:42.955693 (+ 228us) client_negotiation.cc:770] Sending connection context
0501 02:07:42.955928 (+ 235us) client_negotiation.cc:241] Negotiation successful
0501 02:07:42.956136 (+ 208us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":127,"threads_started":1}
I20260501 02:07:42.958026 1159 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:42.958273 1159 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:42.958756 1159 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:42.960346 966 ts_manager.cc:194] Registered new tserver with Master: 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369)
I20260501 02:07:42.961911 966 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:57761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:42.993489 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:43.091768 1166 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:43.092093 1166 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:43.092190 1166 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:43.095536 1166 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:43.095592 1166 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:43.095727 1166 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:07:43.099815 1166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:43.100818 1166 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:43.101609 1166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:43.107908 1174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.107903 1171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.107904 1172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:43.107990 1166 server_base.cc:1061] running on GCE node
I20260501 02:07:43.108573 1166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:43.109079 1166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:43.110227 1166 hybrid_clock.cc:648] HybridClock initialized: now 1777601263110216 us; error 24 us; skew 500 ppm
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:43.113015 1166 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:07:43.113945 1166 webserver.cc:492] Webserver started at http://127.18.96.194:39427/ using document root <none> and password file <none>
I20260501 02:07:43.114467 1166 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:43.114512 1166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:43.114662 1166 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:43.116163 1166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "23a6d60c57234b08973a1a8c0c64047d"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "0d9714ee3abf3773be617f524d8f625a"
server_key_iv: "367c1995a88955c19a00ad869c11d77e"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.116577 1166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "23a6d60c57234b08973a1a8c0c64047d"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "0d9714ee3abf3773be617f524d8f625a"
server_key_iv: "367c1995a88955c19a00ad869c11d77e"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.119642 1166 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:07:43.121476 1181 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.122409 1166 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:43.122529 1166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "23a6d60c57234b08973a1a8c0c64047d"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "0d9714ee3abf3773be617f524d8f625a"
server_key_iv: "367c1995a88955c19a00ad869c11d77e"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.122625 1166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:43.137794 1166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:43.140203 1166 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:43.140388 1166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:43.140908 1166 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:43.141702 1166 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:43.141773 1166 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.141842 1166 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:43.141891 1166 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.150682 1166 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42939
I20260501 02:07:43.150772 1294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42939 every 8 connection(s)
I20260501 02:07:43.151634 1166 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:07:43.158214 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1166
I20260501 02:07:43.158386 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:07:43.158617 18819 external_mini_cluster.cc:1468] Setting key 27bd3ec410951d59944b557867a54870
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:43.163375 1005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.153245 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34797 (local address 127.18.96.254:40969)
0501 02:07:43.153395 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.153399 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.154036 (+ 637us) server_negotiation.cc:408] Connection header received
0501 02:07:43.154829 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.154831 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.154878 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.154951 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:43.156079 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.156606 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.157217 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.157366 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.160161 (+ 2795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.160186 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:43.160188 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:43.160211 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:43.161545 (+ 1334us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.162057 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.162067 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.162069 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.162111 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.162417 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.162429 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.162431 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.162568 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.162680 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.163112 (+ 432us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.163224 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:07:43.163945 1297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.153513 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:34797)
0501 02:07:43.153880 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.153911 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.154669 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.155088 (+ 419us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.155095 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.155477 (+ 382us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:43.155941 (+ 464us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.155960 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.156706 (+ 746us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.156709 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.157124 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.157130 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.157271 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.158404 (+ 1133us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:43.158422 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:43.160021 (+ 1599us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:43.161658 (+ 1637us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.161664 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.161674 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.161922 (+ 248us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.162208 (+ 286us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.162216 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.162217 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.162301 (+ 84us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.162679 (+ 378us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:43.162685 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:43.162880 (+ 195us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.163063 (+ 183us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.163267 (+ 204us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260501 02:07:43.164968 1295 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:43.165216 1295 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:43.165702 1295 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:43.166702 966 ts_manager.cc:194] Registered new tserver with Master: 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939)
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:07:43.167245 966 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:34797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:07:43.207176 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40969
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:43.304888 1302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:43.305087 1302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:43.305187 1302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:43.308449 1302 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:43.308501 1302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:43.308601 1302 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:07:43.312674 1302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:43.313596 1302 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:43.314441 1302 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:43.320331 1310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.320389 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.320418 1308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:43.320623 1302 server_base.cc:1061] running on GCE node
I20260501 02:07:43.320981 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:43.321475 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:43.322647 1302 hybrid_clock.cc:648] HybridClock initialized: now 1777601263322630 us; error 34 us; skew 500 ppm
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:43.325225 1302 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:07:43.326218 1302 webserver.cc:492] Webserver started at http://127.18.96.195:39197/ using document root <none> and password file <none>
I20260501 02:07:43.326731 1302 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:43.326804 1302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:43.327023 1302 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:07:43.328594 1302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "999365ef0dbe4b7ca7593af3b198f22a"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "6b9915c92c159087da7961e8712e4d81"
server_key_iv: "a4334feddddaacf4680cf76652beb669"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.329106 1302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "999365ef0dbe4b7ca7593af3b198f22a"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "6b9915c92c159087da7961e8712e4d81"
server_key_iv: "a4334feddddaacf4680cf76652beb669"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.332167 1302 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:07:43.334129 1317 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.335016 1302 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:43.335140 1302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "999365ef0dbe4b7ca7593af3b198f22a"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "6b9915c92c159087da7961e8712e4d81"
server_key_iv: "a4334feddddaacf4680cf76652beb669"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.335234 1302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:43.346239 1302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:43.348757 1302 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:43.348937 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:43.349424 1302 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:43.350304 1302 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:43.350373 1302 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.350440 1302 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:43.350492 1302 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.359174 1302 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:35139
I20260501 02:07:43.359200 1430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:35139 every 8 connection(s)
I20260501 02:07:43.359987 1302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:07:43.361769 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1302
I20260501 02:07:43.361887 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:07:43.362166 18819 external_mini_cluster.cc:1468] Setting key 41b33fe3063fbaadf0534bc25b0467ab
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:43.371520 1005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.361592 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:41493 (local address 127.18.96.254:40969)
0501 02:07:43.361748 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.361753 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.362504 (+ 751us) server_negotiation.cc:408] Connection header received
0501 02:07:43.363530 (+ 1026us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.363534 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.363592 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.363674 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:43.364854 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.365350 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.365908 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.366108 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.368606 (+ 2498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.368620 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:43.368622 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:43.368659 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:43.370099 (+ 1440us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.370496 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.370498 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.370500 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.370539 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.370762 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.370764 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.370765 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.370882 (+ 117us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.370953 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.371331 (+ 378us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.371410 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:07:43.372042 1433 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.361872 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:41493)
0501 02:07:43.362359 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.362397 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.363327 (+ 930us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.363801 (+ 474us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.363808 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.364304 (+ 496us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:43.364730 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.364740 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.365450 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.365452 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.365817 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.365822 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.366003 (+ 181us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.366951 (+ 948us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:43.366967 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:43.368473 (+ 1506us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:43.370193 (+ 1720us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.370198 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.370205 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.370413 (+ 208us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.370613 (+ 200us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.370615 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.370616 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.370691 (+ 75us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.370961 (+ 270us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:43.370964 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:43.371119 (+ 155us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.371277 (+ 158us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.371444 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260501 02:07:43.373104 1431 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:43.373338 1431 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:43.373777 1431 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:43.374914 966 ts_manager.cc:194] Registered new tserver with Master: 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139)
I20260501 02:07:43.375630 966 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:41493
I20260501 02:07:43.375788 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:07:43.383781 1005 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.377702 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.18.96.254:40969)
0501 02:07:43.377862 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.377866 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.378063 (+ 197us) server_negotiation.cc:408] Connection header received
0501 02:07:43.378146 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.378148 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.378184 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.378257 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:43.379023 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.379456 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.380069 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.380185 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.381196 (+ 1011us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.381209 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:43.381211 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:43.381228 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:43.382477 (+ 1249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.382835 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.382837 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.382838 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.382874 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.383045 (+ 171us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.383047 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.383048 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.383178 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.383272 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.383481 (+ 209us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.383570 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260501 02:07:43.386163 966 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:07:43.388471 966 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:07:43.400594 1444 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.395979 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:35139 (local address 127.0.0.1:57740)
0501 02:07:43.396629 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.396671 (+ 42us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.396829 (+ 158us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.397410 (+ 581us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.397416 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.397439 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.397740 (+ 301us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.397747 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399205 (+ 1458us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.399208 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.400111 (+ 903us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.400119 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.400226 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.400270 (+ 44us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.400364 (+ 94us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.400439 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":71,"threads_started":1}
I20260501 02:07:43.401151 1443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.396102 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.18.96.195:35139)
0501 02:07:43.396777 (+ 675us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.396783 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.396807 (+ 24us) server_negotiation.cc:408] Connection header received
0501 02:07:43.397023 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.397030 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.397213 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.397413 (+ 200us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.397869 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399084 (+ 1215us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.400256 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.400776 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.400888 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.400947 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.401018 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":187,"threads_started":1}
I20260501 02:07:43.401625 1447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.397259 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.18.96.194:42939)
0501 02:07:43.397644 (+ 385us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.397649 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.397734 (+ 85us) server_negotiation.cc:408] Connection header received
0501 02:07:43.397869 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.397876 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.398045 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.398192 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.398552 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399373 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.400665 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.401319 (+ 654us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.401410 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.401460 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.401515 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260501 02:07:43.401793 1445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.396384 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.18.96.193:39369)
0501 02:07:43.397015 (+ 631us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.397022 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.397537 (+ 515us) server_negotiation.cc:408] Connection header received
0501 02:07:43.397625 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.397630 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.397801 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.397942 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.398368 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399469 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.400897 (+ 1428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.401445 (+ 548us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.401533 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.401593 (+ 60us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.401661 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":108,"threads_started":1}
I20260501 02:07:43.400594 1442 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.396983 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42939 (local address 127.0.0.1:57576)
0501 02:07:43.397641 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.397654 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.397740 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.398192 (+ 452us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.398196 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.398219 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.398429 (+ 210us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.398435 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399496 (+ 1061us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.399499 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.400295 (+ 796us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.400302 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.400393 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.400405 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.400450 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.400487 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":81,"threads_started":1}
I20260501 02:07:43.400790 1446 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.396229 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:39369 (local address 127.0.0.1:47144)
0501 02:07:43.397321 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.397334 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.397430 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.397968 (+ 538us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.397971 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.397991 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.398224 (+ 233us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.398230 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.399593 (+ 1363us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.399596 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.400418 (+ 822us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.400427 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.400534 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.400549 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.400598 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.400650 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":102,"threads_started":1}
I20260501 02:07:43.403854 1365 tablet_service.cc:1511] Processing CreateTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (DEFAULT_TABLE table=test-table [id=f2c51d4006a440febb7db7db8f783b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:43.404441 1229 tablet_service.cc:1511] Processing CreateTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (DEFAULT_TABLE table=test-table [id=f2c51d4006a440febb7db7db8f783b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:43.404690 1365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ec53cb7240f43349f1cadefd0b0f6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:43.405156 1229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ec53cb7240f43349f1cadefd0b0f6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:43.409785 1448 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Bootstrap starting.
I20260501 02:07:43.409812 1449 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Bootstrap starting.
I20260501 02:07:43.412107 1448 tablet_bootstrap.cc:654] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:43.412117 1449 tablet_bootstrap.cc:654] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:43.412014 1093 tablet_service.cc:1511] Processing CreateTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (DEFAULT_TABLE table=test-table [id=f2c51d4006a440febb7db7db8f783b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:43.412825 1448 log.cc:826] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:43.412880 1093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ec53cb7240f43349f1cadefd0b0f6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:43.412880 1449 log.cc:826] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:43.414641 1448 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: No bootstrap required, opened a new log
I20260501 02:07:43.414817 1448 ts_tablet_manager.cc:1403] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:07:43.415271 1449 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: No bootstrap required, opened a new log
I20260501 02:07:43.415402 1449 ts_tablet_manager.cc:1403] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:07:43.418041 1449 raft_consensus.cc:359] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.418351 1449 raft_consensus.cc:385] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:43.418303 1448 raft_consensus.cc:359] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.418468 1452 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Bootstrap starting.
I20260501 02:07:43.418478 1449 raft_consensus.cc:740] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a6d60c57234b08973a1a8c0c64047d, State: Initialized, Role: FOLLOWER
I20260501 02:07:43.418608 1448 raft_consensus.cc:385] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:43.418722 1448 raft_consensus.cc:740] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999365ef0dbe4b7ca7593af3b198f22a, State: Initialized, Role: FOLLOWER
I20260501 02:07:43.418926 1449 consensus_queue.cc:260] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.419181 1448 consensus_queue.cc:260] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.419582 1295 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:43.419765 1449 ts_tablet_manager.cc:1434] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:07:43.420034 1448 ts_tablet_manager.cc:1434] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:07:43.420183 1431 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:43.420850 1452 tablet_bootstrap.cc:654] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:43.421646 1452 log.cc:826] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:43.423792 1452 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: No bootstrap required, opened a new log
I20260501 02:07:43.424015 1452 ts_tablet_manager.cc:1403] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:07:43.426929 1452 raft_consensus.cc:359] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.427081 1452 raft_consensus.cc:385] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:43.427163 1452 raft_consensus.cc:740] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec954487e04de09106f2bef8dda8e4, State: Initialized, Role: FOLLOWER
I20260501 02:07:43.427549 1452 consensus_queue.cc:260] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.428207 1159 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:43.428385 1452 ts_tablet_manager.cc:1434] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260501 02:07:43.444561 1160 tablet.cc:2404] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:07:43.496863 1456 raft_consensus.cc:493] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:07:43.497028 1456 raft_consensus.cc:515] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.498134 1456 leader_election.cc:290] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139), 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939)
I20260501 02:07:43.500171 1453 raft_consensus.cc:493] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:07:43.500319 1453 raft_consensus.cc:515] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.501255 1457 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.498318 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42939 (local address 127.18.96.193:35095)
0501 02:07:43.498645 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.498661 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.498776 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.499043 (+ 267us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.499046 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.499064 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.499319 (+ 255us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.499325 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.500098 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.500105 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.500816 (+ 711us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.500825 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.500976 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.500993 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.501048 (+ 55us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.501102 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":27,"thread_start_us":85,"threads_started":1}
I20260501 02:07:43.501433 1453 leader_election.cc:290] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139), 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369)
I20260501 02:07:43.501700 1447 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.498462 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:35095 (local address 127.18.96.194:42939)
0501 02:07:43.498656 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.498661 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.498735 (+ 74us) server_negotiation.cc:408] Connection header received
0501 02:07:43.498890 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.498894 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.498945 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.499030 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.499435 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.499966 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.500940 (+ 974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.501439 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.501483 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.501535 (+ 52us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.501597 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260501 02:07:43.502019 1443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.498458 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:32861 (local address 127.18.96.195:35139)
0501 02:07:43.498614 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.498618 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.498745 (+ 127us) server_negotiation.cc:408] Connection header received
0501 02:07:43.498933 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.498936 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.498982 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.499061 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.499470 (+ 409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.499963 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.501065 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.501537 (+ 472us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.501574 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.501788 (+ 214us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.501847 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:07:43.502353 1458 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.498318 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:35139 (local address 127.18.96.193:32861)
0501 02:07:43.498644 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.498665 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.498816 (+ 151us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.499083 (+ 267us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.499086 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.499109 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.499359 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.499364 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.500071 (+ 707us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.500078 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.500925 (+ 847us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.500934 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.501640 (+ 706us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.501659 (+ 19us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.502141 (+ 482us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.502195 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"spinlock_wait_cycles":5248,"thread_start_us":73,"threads_started":1}
I20260501 02:07:43.502341 1249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a6d60c57234b08973a1a8c0c64047d" is_pre_election: true
I20260501 02:07:43.502521 1249 raft_consensus.cc:2468] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 0.
I20260501 02:07:43.503121 1047 leader_election.cc:304] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a6d60c57234b08973a1a8c0c64047d, 77ec954487e04de09106f2bef8dda8e4; no voters:
I20260501 02:07:43.503383 1456 raft_consensus.cc:2804] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:07:43.503460 1456 raft_consensus.cc:493] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:07:43.503515 1456 raft_consensus.cc:3060] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:43.504441 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999365ef0dbe4b7ca7593af3b198f22a" is_pre_election: true
I20260501 02:07:43.504739 1456 raft_consensus.cc:515] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.504779 1385 raft_consensus.cc:2468] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 0.
I20260501 02:07:43.505097 1456 leader_election.cc:290] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 election: Requested vote from peers 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139), 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939)
I20260501 02:07:43.505612 1385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999365ef0dbe4b7ca7593af3b198f22a"
I20260501 02:07:43.505797 1385 raft_consensus.cc:3060] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:43.505846 1249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a6d60c57234b08973a1a8c0c64047d"
I20260501 02:07:43.505947 1249 raft_consensus.cc:3060] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:43.506522 1443 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.502830 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40801 (local address 127.18.96.195:35139)
0501 02:07:43.502975 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.502981 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.503155 (+ 174us) server_negotiation.cc:408] Connection header received
0501 02:07:43.503252 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.503258 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.503315 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.503398 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.503928 (+ 530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.504594 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.505726 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.506155 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.506184 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.506378 (+ 194us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.506425 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260501 02:07:43.506877 1459 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.502694 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:35139 (local address 127.18.96.194:40801)
0501 02:07:43.503058 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.503071 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.503166 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.503486 (+ 320us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.503490 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.503507 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.503797 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.503803 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.504731 (+ 928us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.504734 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.505592 (+ 858us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.505601 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.506242 (+ 641us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.506255 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.506703 (+ 448us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.506760 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260501 02:07:43.507153 1249 raft_consensus.cc:2468] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 1.
I20260501 02:07:43.507453 1047 leader_election.cc:304] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a6d60c57234b08973a1a8c0c64047d, 77ec954487e04de09106f2bef8dda8e4; no voters:
I20260501 02:07:43.507505 1385 raft_consensus.cc:2468] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 1.
I20260501 02:07:43.507601 1384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "23a6d60c57234b08973a1a8c0c64047d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999365ef0dbe4b7ca7593af3b198f22a" is_pre_election: true
I20260501 02:07:43.507766 1384 raft_consensus.cc:2393] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 23a6d60c57234b08973a1a8c0c64047d in current term 1: Already voted for candidate 77ec954487e04de09106f2bef8dda8e4 in this term.
I20260501 02:07:43.508021 1456 raft_consensus.cc:2804] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:43.512228 1456 raft_consensus.cc:697] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 77ec954487e04de09106f2bef8dda8e4, State: Running, Role: LEADER
I20260501 02:07:43.512413 1297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.501776 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:39369 (local address 127.18.96.194:40531)
0501 02:07:43.501910 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.501928 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.502092 (+ 164us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.510045 (+ 7953us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.510048 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.510070 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.510260 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.510265 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.511242 (+ 977us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.511246 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.512078 (+ 832us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.512087 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.512188 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.512202 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.512249 (+ 47us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.512301 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260501 02:07:43.512552 1456 consensus_queue.cc:237] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.512898 1445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.502218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:40531 (local address 127.18.96.193:39369)
0501 02:07:43.502354 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.502361 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.502376 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:07:43.502412 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.502414 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.503742 (+ 1328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.503933 (+ 191us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.510390 (+ 6457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.511122 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.512229 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.512686 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.512715 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.512761 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.512806 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:07:43.513386 1113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ec53cb7240f43349f1cadefd0b0f6d7" candidate_uuid: "23a6d60c57234b08973a1a8c0c64047d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ec954487e04de09106f2bef8dda8e4" is_pre_election: true
I20260501 02:07:43.513948 1184 leader_election.cc:304] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23a6d60c57234b08973a1a8c0c64047d; no voters: 77ec954487e04de09106f2bef8dda8e4, 999365ef0dbe4b7ca7593af3b198f22a
I20260501 02:07:43.514184 1453 raft_consensus.cc:2749] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 02:07:43.515959 965 catalog_manager.cc:5671] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "77ec954487e04de09106f2bef8dda8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } health_report { overall_health: UNKNOWN } } }
I20260501 02:07:43.554725 1445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.551752 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.18.96.193:39369)
0501 02:07:43.551882 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.551886 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.551899 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:07:43.551983 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.551985 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.552034 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.552111 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:07:43.552457 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.552918 (+ 461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.553602 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.553756 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.553835 (+ 79us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:43.554207 (+ 372us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:43.554305 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.554444 (+ 139us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.554539 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:07:43.560042 1384 raft_consensus.cc:1275] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Refusing update from remote peer 77ec954487e04de09106f2bef8dda8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:43.560070 1249 raft_consensus.cc:1275] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Refusing update from remote peer 77ec954487e04de09106f2bef8dda8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:43.560691 1456 consensus_queue.cc:1048] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:43.560883 1460 consensus_queue.cc:1048] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:43.569815 1464 mvcc.cc:204] Tried to move back new op lower bound from 7281054775535370240 to 7281054775350251520. Current Snapshot: MvccSnapshot[applied={T|T < 7281054775535370240 or (T in {7281054775535370240})}]
I20260501 02:07:43.575481 966 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:43.575610 966 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:43.577936 966 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Sending DeleteTablet for 3 replicas of tablet 3ec53cb7240f43349f1cadefd0b0f6d7
I20260501 02:07:43.578653 1093 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:43 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:47144
I20260501 02:07:43.578832 1365 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:43 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:57740
I20260501 02:07:43.578847 1229 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:43 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:57576
I20260501 02:07:43.579234 1474 tablet_replica.cc:333] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: stopping tablet replica
I20260501 02:07:43.579470 1473 tablet_replica.cc:333] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: stopping tablet replica
I20260501 02:07:43.579468 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1030
I20260501 02:07:43.579517 1474 raft_consensus.cc:2243] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:43.580298 1474 raft_consensus.cc:2272] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:43.580757 1475 tablet_replica.cc:333] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: stopping tablet replica
I20260501 02:07:43.581007 1475 raft_consensus.cc:2243] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:43.581279 1475 raft_consensus.cc:2272] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:43.582398 1474 ts_tablet_manager.cc:1916] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:43.582943 1475 ts_tablet_manager.cc:1916] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:43.584877 1475 ts_tablet_manager.cc:1929] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:43.584981 1475 log.cc:1199] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/3ec53cb7240f43349f1cadefd0b0f6d7
I20260501 02:07:43.585543 1474 ts_tablet_manager.cc:1929] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:43.585548 1475 ts_tablet_manager.cc:1950] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 23a6d60c57234b08973a1a8c0c64047d: Deleting consensus metadata
I20260501 02:07:43.585668 1474 log.cc:1199] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/3ec53cb7240f43349f1cadefd0b0f6d7
I20260501 02:07:43.586031 1474 ts_tablet_manager.cc:1950] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 999365ef0dbe4b7ca7593af3b198f22a: Deleting consensus metadata
I20260501 02:07:43.586974 951 catalog_manager.cc:5002] TS 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939): tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (table test-table [id=f2c51d4006a440febb7db7db8f783b12]) successfully deleted
I20260501 02:07:43.587103 950 catalog_manager.cc:5002] TS 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139): tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (table test-table [id=f2c51d4006a440febb7db7db8f783b12]) successfully deleted
W20260501 02:07:43.587643 952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:39369 (error 108)
I20260501 02:07:43.587903 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1166
I20260501 02:07:43.589000 1446 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.588761 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:39369 (local address 127.0.0.1:47158)
0501 02:07:43.588885 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.588935 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:39369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260501 02:07:43.589210 952 catalog_manager.cc:4729] TS 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ec53cb7240f43349f1cadefd0b0f6d7: Network error: Client connection negotiation failed: client connection to 127.18.96.193:39369: connect: Connection refused (error 111)
I20260501 02:07:43.594561 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1302
I20260501 02:07:43.600277 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 934
I20260501 02:07:43.606773 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--webserver_interface=127.18.96.254
--webserver_port=43137
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:43.704941 1477 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:43.705145 1477 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:43.705235 1477 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:43.708519 1477 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:43.708582 1477 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:43.708606 1477 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:43.708626 1477 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:43.708689 1477 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:43.712831 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=43137
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:43.713833 1477 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:43.714668 1477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:43.723938 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.723938 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:43.724031 1477 server_base.cc:1061] running on GCE node
W20260501 02:07:43.723969 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:43.724769 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:43.725621 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:43.726823 1477 hybrid_clock.cc:648] HybridClock initialized: now 1777601263726802 us; error 40 us; skew 500 ppm
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:43.729569 1477 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:43.730665 1477 webserver.cc:492] Webserver started at http://127.18.96.254:43137/ using document root <none> and password file <none>
I20260501 02:07:43.731333 1477 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:43.731418 1477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:43.734704 1477 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:07:43.736416 1493 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.737403 1477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:43.737536 1477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "55d1676472604eb8bcddebac42c42215"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "e74ed1f6ecbce5080f995726ab2f2d79"
server_key_iv: "feecc70b5c443617954cc0831bcaef1f"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.737944 1477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:43.749015 1477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:43.751302 1477 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:43.751507 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:43.758512 1477 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40969
I20260501 02:07:43.758517 1545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40969 every 8 connection(s)
I20260501 02:07:43.759554 1477 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:43.761731 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1477
I20260501 02:07:43.762226 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:39369
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=34277
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:43.763783 1546 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:07:43.765828 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap starting.
I20260501 02:07:43.781898 1546 log.cc:826] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:43.789746 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:07:43.790159 1546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap complete.
I20260501 02:07:43.793439 1546 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:43.793756 1546 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Initialized, Role: FOLLOWER
I20260501 02:07:43.794231 1546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:43.794391 1546 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:43.794473 1546 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:43.794591 1546 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:07:43.795946 1546 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:43.796284 1546 leader_election.cc:304] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d1676472604eb8bcddebac42c42215; no voters:
I20260501 02:07:43.796579 1546 leader_election.cc:290] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 02:07:43.796658 1550 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Leader election won for term 2
I20260501 02:07:43.796854 1550 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 LEADER]: Becoming Leader. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Running, Role: LEADER
I20260501 02:07:43.797485 1550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:43.797653 1546 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:43.798869 1552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d1676472604eb8bcddebac42c42215. Latest consensus state: current_term: 2 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:43.798879 1551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:43.798996 1552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:43.799006 1551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:43.799357 1559 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:43.803824 1559 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:43.804512 1559 catalog_manager.cc:1269] Loaded cluster ID: 0108d21bfea14acc82b6063388e2021f
I20260501 02:07:43.804590 1559 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:07:43.805876 1559 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260501 02:07:43.808784 1559 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20260501 02:07:43.813778 1559 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:07:43.814505 1559 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Loaded TSK: 0
I20260501 02:07:43.815438 1559 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:07:43.867069 1548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:43.867244 1548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:43.867331 1548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:43.870575 1548 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:43.870625 1548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:43.870783 1548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:07:43.874681 1548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:39369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=34277
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:43.875555 1548 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:43.876272 1548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:43.882215 1574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.882273 1576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:43.882210 1573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:43.882714 1548 server_base.cc:1061] running on GCE node
I20260501 02:07:43.883088 1548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:43.883677 1548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:43.884855 1548 hybrid_clock.cc:648] HybridClock initialized: now 1777601263884825 us; error 37 us; skew 500 ppm
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:43.888080 1548 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:07:43.889199 1548 webserver.cc:492] Webserver started at http://127.18.96.193:34277/ using document root <none> and password file <none>
I20260501 02:07:43.889858 1548 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:43.889930 1548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:43.894029 1548 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:07:43.896168 1583 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:43.897265 1548 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:43.897368 1548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "77ec954487e04de09106f2bef8dda8e4"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "69cdfd91847681dc6c20a14dd682d2f7"
server_key_iv: "3164baff08ad191c1f3fbde45798233e"
server_key_version: "encryptionkey@0"
I20260501 02:07:43.897800 1548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:43.932225 1548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:43.935490 1548 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:43.935704 1548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:43.936372 1548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:43.937726 1590 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 02:07:43.941431 1548 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 02:07:43.941504 1548 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260501 02:07:43.941576 1548 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 02:07:43.943569 1548 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 02:07:43.943670 1548 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:43.944022 1590 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Bootstrap starting.
I20260501 02:07:43.955116 1548 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:39369
I20260501 02:07:43.955348 1697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:39369 every 8 connection(s)
I20260501 02:07:43.956390 1548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:07:43.960089 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1548
I20260501 02:07:43.960573 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:42939
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=39427
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:43.962869 1590 log.cc:826] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Log is configured to *not* fsync() on all Append() calls
May 01 02:07:43 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601263, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:43.971637 1590 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:07:43.972153 1590 tablet_bootstrap.cc:492] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Bootstrap complete.
I20260501 02:07:43.973114 1701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.958424 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:59793)
0501 02:07:43.959136 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.959177 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.960195 (+ 1018us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.961812 (+ 1617us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.961820 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.962335 (+ 515us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:43.962939 (+ 604us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.962955 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.964585 (+ 1630us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.964589 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.965178 (+ 589us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.965189 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.965367 (+ 178us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.965990 (+ 623us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:43.966013 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:43.967819 (+ 1806us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:43.970372 (+ 2553us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.970378 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.970389 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.970683 (+ 294us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.970988 (+ 305us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:43.970991 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:43.970993 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:43.971111 (+ 118us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:43.971688 (+ 577us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:43.971693 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:43.971901 (+ 208us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.972098 (+ 197us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.972306 (+ 208us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":125,"threads_started":1}
I20260501 02:07:43.973659 1590 ts_tablet_manager.cc:1403] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260501 02:07:43.974484 1700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.958473 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:59793 (local address 127.18.96.254:40969)
0501 02:07:43.958788 (+ 315us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.958798 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.959322 (+ 524us) server_negotiation.cc:408] Connection header received
0501 02:07:43.960682 (+ 1360us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.960704 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.961032 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.961339 (+ 307us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:43.963150 (+ 1811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.964434 (+ 1284us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.965312 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.965574 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.967995 (+ 2421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:43.968035 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:43.968046 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:43.968078 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:43.970227 (+ 2149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.970809 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.970814 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.970819 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.970896 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:43.971206 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:43.971209 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:43.971211 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:43.971508 (+ 297us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:43.971635 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.973660 (+ 2025us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.973867 (+ 207us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":109,"threads_started":1}
I20260501 02:07:43.976692 1698 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:43.976902 1698 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:43.976994 1590 raft_consensus.cc:359] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.977440 1698 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:43.977495 1590 raft_consensus.cc:740] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec954487e04de09106f2bef8dda8e4, State: Initialized, Role: FOLLOWER
I20260501 02:07:43.977998 1590 consensus_queue.cc:260] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } } peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } }
I20260501 02:07:43.978947 1590 ts_tablet_manager.cc:1434] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:07:43.980625 1510 ts_manager.cc:194] Registered new tserver with Master: 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369)
I20260501 02:07:43.984299 1510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:59793
I20260501 02:07:43.986433 1698 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:43.989248 1705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.983661 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:39369 (local address 127.0.0.1:47164)
0501 02:07:43.984673 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:43.984705 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:43.984822 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:43.986387 (+ 1565us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:43.986392 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:43.986413 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:43.986676 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.986682 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.987980 (+ 1298us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.987984 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:43.988840 (+ 856us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:43.988849 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.988953 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.988985 (+ 32us) client_negotiation.cc:770] Sending connection context
0501 02:07:43.989053 (+ 68us) client_negotiation.cc:241] Negotiation successful
0501 02:07:43.989114 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":102,"threads_started":1}
I20260501 02:07:43.989811 1706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:43.984298 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.18.96.193:39369)
0501 02:07:43.985051 (+ 753us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:43.985056 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:43.985074 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:07:43.985133 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:43.985138 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:43.986131 (+ 993us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:43.986261 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:43.986803 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.987850 (+ 1047us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:43.988971 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:43.989459 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:43.989546 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:43.989612 (+ 66us) server_negotiation.cc:300] Negotiation successful
0501 02:07:43.989677 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":87,"threads_started":1}
I20260501 02:07:43.990603 1632 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ec53cb7240f43349f1cadefd0b0f6d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:43 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:47164
I20260501 02:07:43.991406 1707 tablet_replica.cc:333] stopping tablet replica
I20260501 02:07:43.991647 1707 raft_consensus.cc:2243] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:43.991882 1707 raft_consensus.cc:2272] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:43.992862 1707 ts_tablet_manager.cc:1916] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:43.996304 1707 ts_tablet_manager.cc:1929] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:43.996389 1707 log.cc:1199] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/3ec53cb7240f43349f1cadefd0b0f6d7
I20260501 02:07:43.996687 1707 ts_tablet_manager.cc:1950] T 3ec53cb7240f43349f1cadefd0b0f6d7 P 77ec954487e04de09106f2bef8dda8e4: Deleting consensus metadata
I20260501 02:07:43.998193 1496 catalog_manager.cc:5002] TS 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369): tablet 3ec53cb7240f43349f1cadefd0b0f6d7 (table test-table [id=f2c51d4006a440febb7db7db8f783b12]) successfully deleted
W20260501 02:07:44.069746 1702 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:44.070021 1702 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:44.070134 1702 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:44.073436 1702 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:44.073523 1702 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:44.073635 1702 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:07:44.077857 1702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:42939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=39427
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:44.078830 1702 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:44.079638 1702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:44.085804 1712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.085819 1715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.085819 1713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.086098 1702 server_base.cc:1061] running on GCE node
I20260501 02:07:44.086480 1702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:44.086999 1702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:44.088143 1702 hybrid_clock.cc:648] HybridClock initialized: now 1777601264088128 us; error 28 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:44.090703 1702 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:07:44.091707 1702 webserver.cc:492] Webserver started at http://127.18.96.194:39427/ using document root <none> and password file <none>
I20260501 02:07:44.092213 1702 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:44.092305 1702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:44.095400 1702 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:07:44.097031 1722 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.098068 1702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:07:44.098171 1702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "23a6d60c57234b08973a1a8c0c64047d"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "0d9714ee3abf3773be617f524d8f625a"
server_key_iv: "367c1995a88955c19a00ad869c11d77e"
server_key_version: "encryptionkey@0"
I20260501 02:07:44.098584 1702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:44.116849 1702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:44.119151 1702 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:44.119345 1702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:44.119868 1702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:44.120690 1702 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:44.120734 1702 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.120800 1702 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:44.120838 1702 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.129761 1702 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42939
I20260501 02:07:44.129851 1835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42939 every 8 connection(s)
I20260501 02:07:44.130759 1702 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:44.140524 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1702
I20260501 02:07:44.140980 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:35139
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=39197
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:44.141757 1700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.132275 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46499 (local address 127.18.96.254:40969)
0501 02:07:44.132442 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.132446 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.132989 (+ 543us) server_negotiation.cc:408] Connection header received
0501 02:07:44.133704 (+ 715us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.133707 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.133758 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.133830 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:44.134827 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.135320 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.135893 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.136019 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.138438 (+ 2419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:44.138457 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:44.138459 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:44.138481 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:44.139698 (+ 1217us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.140090 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.140092 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.140093 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.140130 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.140542 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.140548 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.140551 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.140721 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:44.140825 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.141339 (+ 514us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.141481 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:07:44.142150 1838 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.132521 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:46499)
0501 02:07:44.132870 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.132897 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.133547 (+ 650us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.133975 (+ 428us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.133982 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.134317 (+ 335us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:44.134710 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.134719 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.135439 (+ 720us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.135441 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.135802 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.135807 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.135941 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.136915 (+ 974us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:44.136931 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:44.138282 (+ 1351us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:44.139799 (+ 1517us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:44.139803 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:44.139811 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:44.140007 (+ 196us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:44.140281 (+ 274us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:44.140293 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:44.140298 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:44.140420 (+ 122us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:44.140822 (+ 402us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:44.140827 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:44.140990 (+ 163us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.141245 (+ 255us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.141442 (+ 197us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":89,"threads_started":1}
I20260501 02:07:44.143307 1836 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:44.143543 1836 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:44.144080 1836 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:44.145197 1510 ts_manager.cc:194] Registered new tserver with Master: 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939)
I20260501 02:07:44.145740 1510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:46499
W20260501 02:07:44.237964 1839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:44.238212 1839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:44.238298 1839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:44.241506 1839 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:44.241560 1839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:44.241683 1839 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:07:44.245857 1839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:35139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=39197
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:44.246860 1839 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:44.247583 1839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:44.253598 1844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.253645 1845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.253645 1847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.253897 1839 server_base.cc:1061] running on GCE node
I20260501 02:07:44.254253 1839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:44.254762 1839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:44.255941 1839 hybrid_clock.cc:648] HybridClock initialized: now 1777601264255932 us; error 33 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:44.258577 1839 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:07:44.259410 1839 webserver.cc:492] Webserver started at http://127.18.96.195:39197/ using document root <none> and password file <none>
I20260501 02:07:44.259893 1839 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:44.259945 1839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:44.262959 1839 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:07:44.264595 1854 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.265492 1839 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:44.265578 1839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "999365ef0dbe4b7ca7593af3b198f22a"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "6b9915c92c159087da7961e8712e4d81"
server_key_iv: "a4334feddddaacf4680cf76652beb669"
server_key_version: "encryptionkey@0"
I20260501 02:07:44.265929 1839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:44.283551 1839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:44.286059 1839 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:44.286201 1839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:44.286720 1839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:44.287566 1839 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:44.287611 1839 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.287647 1839 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:44.287662 1839 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.298103 1839 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:35139
I20260501 02:07:44.298166 1967 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:35139 every 8 connection(s)
I20260501 02:07:44.299199 1839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:07:44.307204 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 1839
I20260501 02:07:44.311699 1700 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.300831 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:42737 (local address 127.18.96.254:40969)
0501 02:07:44.300968 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.300972 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.301594 (+ 622us) server_negotiation.cc:408] Connection header received
0501 02:07:44.302477 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.302480 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.302535 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.302607 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:44.303755 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.304259 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.304892 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.305066 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.307823 (+ 2757us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:44.307841 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:44.307844 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:44.307873 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:44.309641 (+ 1768us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.310205 (+ 564us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.310208 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.310210 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.310251 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.310582 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.310584 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.310585 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.310756 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:44.310846 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.311357 (+ 511us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.311476 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:07:44.312240 1970 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.301094 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:42737)
0501 02:07:44.301465 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.301495 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.302297 (+ 802us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.302761 (+ 464us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.302768 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.303193 (+ 425us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:07:44.303634 (+ 441us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.303645 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.304397 (+ 752us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.304402 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.304775 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.304781 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.304976 (+ 195us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.306043 (+ 1067us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:07:44.306061 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:07:44.307633 (+ 1572us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:07:44.309763 (+ 2130us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:44.309769 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:44.309779 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:44.310113 (+ 334us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:44.310374 (+ 261us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:07:44.310378 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:07:44.310380 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:07:44.310483 (+ 103us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:07:44.310872 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:07:44.310877 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:07:44.311088 (+ 211us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.311257 (+ 169us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.311511 (+ 254us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
I20260501 02:07:44.313246 1968 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40969
I20260501 02:07:44.313488 1968 heartbeater.cc:461] Registering TS with master...
I20260501 02:07:44.314093 1968 heartbeater.cc:507] Master 127.18.96.254:40969 requested a full tablet report, sending...
I20260501 02:07:44.315284 1510 ts_manager.cc:194] Registered new tserver with Master: 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139)
I20260501 02:07:44.315865 1971 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.307871 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.18.96.254:40969)
0501 02:07:44.308098 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.308101 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.308359 (+ 258us) server_negotiation.cc:408] Connection header received
0501 02:07:44.308497 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.308500 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.308549 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.308624 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:07:44.309763 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.310579 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.311359 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.311698 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.312718 (+ 1020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:07:44.312738 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:07:44.312760 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:07:44.312783 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:07:44.314424 (+ 1641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.314785 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.314789 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.314790 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.314844 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:07:44.315089 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:07:44.315092 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:07:44.315093 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:07:44.315235 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:07:44.315425 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.315616 (+ 191us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.315720 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
I20260501 02:07:44.315995 1510 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:42737
I20260501 02:07:44.317498 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:07:44.320701 1971 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.318159 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.18.96.254:40969)
0501 02:07:44.318319 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.318322 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.318510 (+ 188us) server_negotiation.cc:408] Connection header received
0501 02:07:44.318614 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.318617 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.318675 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.318744 (+ 69us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:07:44.319026 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.319456 (+ 430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.319997 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.320106 (+ 109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.320161 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:44.320426 (+ 265us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:44.320501 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.320595 (+ 94us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.320629 (+ 34us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:07:44.321139 1509 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:07:44.323634 1509 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:07:44.330595 1632 tablet_service.cc:1511] Processing CreateTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef (DEFAULT_TABLE table=test-table [id=e9f05fe847ea499e9a5aab35180e1798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:44.331179 1632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8925bce2f75b48e4b436dcc35503c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:44.333310 1705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.329439 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42939 (local address 127.0.0.1:57590)
0501 02:07:44.329658 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.329672 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.329738 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.330617 (+ 879us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.330622 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.330642 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:44.330830 (+ 188us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.330837 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.332100 (+ 1263us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.332103 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.332977 (+ 874us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.332986 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.333098 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.333112 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.333157 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.333206 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260501 02:07:44.333590 1978 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.329869 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:35139 (local address 127.0.0.1:57748)
0501 02:07:44.330096 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.330109 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.330207 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.331044 (+ 837us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.331047 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.331065 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:44.331262 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.331268 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.332579 (+ 1311us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.332585 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.333232 (+ 647us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.333241 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.333357 (+ 116us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.333371 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.333428 (+ 57us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.333475 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260501 02:07:44.333901 1590 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Bootstrap starting.
I20260501 02:07:44.334206 1977 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.329616 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.18.96.194:42939)
0501 02:07:44.330135 (+ 519us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.330143 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.330184 (+ 41us) server_negotiation.cc:408] Connection header received
0501 02:07:44.330270 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.330277 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.330468 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.330613 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:44.330961 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.331995 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.333125 (+ 1130us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.333821 (+ 696us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.333924 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.334014 (+ 90us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.334083 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":118,"threads_started":1}
I20260501 02:07:44.334626 1979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.330252 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.18.96.195:35139)
0501 02:07:44.330608 (+ 356us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.330613 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.330630 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:07:44.330695 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.330698 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.330836 (+ 138us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.331048 (+ 212us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:44.331402 (+ 354us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.332463 (+ 1061us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.333846 (+ 1383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.334336 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.334428 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.334483 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.334533 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":142,"threads_started":1}
I20260501 02:07:44.334859 1590 tablet_bootstrap.cc:654] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:44.335904 1590 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: No bootstrap required, opened a new log
I20260501 02:07:44.336004 1590 ts_tablet_manager.cc:1403] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:44.336410 1590 raft_consensus.cc:359] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.336522 1590 raft_consensus.cc:385] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:44.336568 1590 raft_consensus.cc:740] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ec954487e04de09106f2bef8dda8e4, State: Initialized, Role: FOLLOWER
I20260501 02:07:44.336688 1590 consensus_queue.cc:260] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.336935 1590 ts_tablet_manager.cc:1434] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:07:44.337044 1902 tablet_service.cc:1511] Processing CreateTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef (DEFAULT_TABLE table=test-table [id=e9f05fe847ea499e9a5aab35180e1798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:44.337028 1770 tablet_service.cc:1511] Processing CreateTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef (DEFAULT_TABLE table=test-table [id=e9f05fe847ea499e9a5aab35180e1798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:07:44.337852 1902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8925bce2f75b48e4b436dcc35503c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:44.337868 1770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8925bce2f75b48e4b436dcc35503c9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:07:44.343617 1982 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Bootstrap starting.
I20260501 02:07:44.343665 1981 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Bootstrap starting.
I20260501 02:07:44.345355 1981 tablet_bootstrap.cc:654] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:44.345355 1982 tablet_bootstrap.cc:654] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Neither blocks nor log segments found. Creating new log.
I20260501 02:07:44.346057 1981 log.cc:826] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:44.346057 1982 log.cc:826] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:44.347611 1982 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: No bootstrap required, opened a new log
I20260501 02:07:44.347667 1981 tablet_bootstrap.cc:492] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: No bootstrap required, opened a new log
I20260501 02:07:44.347803 1982 ts_tablet_manager.cc:1403] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:07:44.348042 1981 ts_tablet_manager.cc:1403] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:07:44.350713 1982 raft_consensus.cc:359] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.350725 1981 raft_consensus.cc:359] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.350901 1982 raft_consensus.cc:385] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:44.350894 1981 raft_consensus.cc:385] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:07:44.350989 1981 raft_consensus.cc:740] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999365ef0dbe4b7ca7593af3b198f22a, State: Initialized, Role: FOLLOWER
I20260501 02:07:44.350986 1982 raft_consensus.cc:740] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a6d60c57234b08973a1a8c0c64047d, State: Initialized, Role: FOLLOWER
I20260501 02:07:44.351357 1982 consensus_queue.cc:260] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.351379 1981 consensus_queue.cc:260] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.351944 1836 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:44.351945 1968 heartbeater.cc:499] Master 127.18.96.254:40969 was elected leader, sending a full tablet report...
I20260501 02:07:44.352151 1981 ts_tablet_manager.cc:1434] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:07:44.352887 1982 ts_tablet_manager.cc:1434] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260501 02:07:44.381560 1837 tablet.cc:2404] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:07:44.397434 1704 raft_consensus.cc:493] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:07:44.397648 1704 raft_consensus.cc:515] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.398808 1704 leader_election.cc:290] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939), 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139)
I20260501 02:07:44.401566 1701 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.399005 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42939 (local address 127.18.96.193:50497)
0501 02:07:44.399147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.399162 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.399254 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.399631 (+ 377us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.399634 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.399697 (+ 63us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:44.399877 (+ 180us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.399882 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.400626 (+ 744us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.400631 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.401179 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.401185 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.401312 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.401329 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.401381 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.401431 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260501 02:07:44.401789 1987 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.399087 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:35139 (local address 127.18.96.193:58385)
0501 02:07:44.399443 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.399455 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.399578 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.399896 (+ 318us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:07:44.399899 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:07:44.399917 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:07:44.400107 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.400113 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.400869 (+ 756us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.400873 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:07:44.401408 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:07:44.401414 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.401549 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.401564 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:07:44.401616 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:07:44.401665 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":84,"threads_started":1}
I20260501 02:07:44.402060 1977 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.399187 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:50497 (local address 127.18.96.194:42939)
0501 02:07:44.399351 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.399355 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.399372 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:07:44.399425 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.399428 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.399489 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.399567 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:44.400001 (+ 434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.400501 (+ 500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.401298 (+ 797us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.401773 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.401818 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.401877 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.401935 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:07:44.402311 1979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.399203 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:58385 (local address 127.18.96.195:35139)
0501 02:07:44.399367 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.399370 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.399593 (+ 223us) server_negotiation.cc:408] Connection header received
0501 02:07:44.399750 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.399753 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.399792 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.399873 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:07:44.400234 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.400712 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.401527 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.402047 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.402077 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.402117 (+ 40us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.402190 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:07:44.402778 1790 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8925bce2f75b48e4b436dcc35503c9ef" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a6d60c57234b08973a1a8c0c64047d" is_pre_election: true
I20260501 02:07:44.402900 1922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8925bce2f75b48e4b436dcc35503c9ef" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999365ef0dbe4b7ca7593af3b198f22a" is_pre_election: true
I20260501 02:07:44.403059 1790 raft_consensus.cc:2468] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 0.
I20260501 02:07:44.403187 1922 raft_consensus.cc:2468] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 0.
I20260501 02:07:44.403589 1585 leader_election.cc:304] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23a6d60c57234b08973a1a8c0c64047d, 77ec954487e04de09106f2bef8dda8e4; no voters:
I20260501 02:07:44.403853 1704 raft_consensus.cc:2804] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:07:44.403930 1704 raft_consensus.cc:493] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:07:44.403978 1704 raft_consensus.cc:3060] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:44.405170 1704 raft_consensus.cc:515] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.405534 1704 leader_election.cc:290] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 election: Requested vote from peers 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939), 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139)
I20260501 02:07:44.405853 1922 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8925bce2f75b48e4b436dcc35503c9ef" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999365ef0dbe4b7ca7593af3b198f22a"
I20260501 02:07:44.405875 1790 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8925bce2f75b48e4b436dcc35503c9ef" candidate_uuid: "77ec954487e04de09106f2bef8dda8e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23a6d60c57234b08973a1a8c0c64047d"
I20260501 02:07:44.405997 1922 raft_consensus.cc:3060] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:44.406021 1790 raft_consensus.cc:3060] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:07:44.406915 1922 raft_consensus.cc:2468] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 1.
I20260501 02:07:44.406931 1790 raft_consensus.cc:2468] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ec954487e04de09106f2bef8dda8e4 in term 1.
I20260501 02:07:44.407239 1584 leader_election.cc:304] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ec954487e04de09106f2bef8dda8e4, 999365ef0dbe4b7ca7593af3b198f22a; no voters:
I20260501 02:07:44.407449 1704 raft_consensus.cc:2804] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:07:44.407688 1704 raft_consensus.cc:697] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 77ec954487e04de09106f2bef8dda8e4, State: Running, Role: LEADER
I20260501 02:07:44.407943 1704 consensus_queue.cc:237] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } }
I20260501 02:07:44.410692 1510 catalog_manager.cc:5671] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "77ec954487e04de09106f2bef8dda8e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ec954487e04de09106f2bef8dda8e4" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 39369 } health_report { overall_health: HEALTHY } } }
I20260501 02:07:44.434321 1706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.431630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.18.96.193:39369)
0501 02:07:44.431757 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:07:44.431760 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:07:44.431772 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:07:44.431842 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:07:44.431844 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:07:44.431913 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:07:44.431995 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:07:44.432311 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.432805 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:07:44.433382 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:07:44.433532 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:07:44.433575 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:07:44.433924 (+ 349us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:07:44.434077 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
0501 02:07:44.434170 (+ 93us) server_negotiation.cc:300] Negotiation successful
0501 02:07:44.434211 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260501 02:07:44.438068 1922 raft_consensus.cc:1275] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Refusing update from remote peer 77ec954487e04de09106f2bef8dda8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:44.438068 1790 raft_consensus.cc:1275] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Refusing update from remote peer 77ec954487e04de09106f2bef8dda8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:07:44.438688 1704 consensus_queue.cc:1048] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999365ef0dbe4b7ca7593af3b198f22a" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 35139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:44.438897 1988 consensus_queue.cc:1048] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a6d60c57234b08973a1a8c0c64047d" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:07:44.448028 1990 mvcc.cc:204] Tried to move back new op lower bound from 7281054779131830272 to 7281054779017478144. Current Snapshot: MvccSnapshot[applied={T|T < 7281054779131830272 or (T in {7281054779131830272})}]
I20260501 02:07:44.452385 1510 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:44.452541 1510 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:07:44.454916 1510 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Sending DeleteTablet for 3 replicas of tablet 8925bce2f75b48e4b436dcc35503c9ef
I20260501 02:07:44.455560 1770 tablet_service.cc:1558] Processing DeleteTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:44 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:57590
I20260501 02:07:44.455605 1902 tablet_service.cc:1558] Processing DeleteTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:44 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:57748
I20260501 02:07:44.455729 1632 tablet_service.cc:1558] Processing DeleteTablet for tablet 8925bce2f75b48e4b436dcc35503c9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:07:44 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:47164
I20260501 02:07:44.455888 1707 tablet_replica.cc:333] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: stopping tablet replica
I20260501 02:07:44.455998 1707 raft_consensus.cc:2243] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:07:44.456058 2000 tablet_replica.cc:333] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: stopping tablet replica
I20260501 02:07:44.456240 1707 raft_consensus.cc:2272] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:44.456292 2000 raft_consensus.cc:2243] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:44.456578 2000 raft_consensus.cc:2272] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:44.456911 1999 tablet_replica.cc:333] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: stopping tablet replica
I20260501 02:07:44.457146 1999 raft_consensus.cc:2243] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:07:44.457295 1707 ts_tablet_manager.cc:1916] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:44.457387 1999 raft_consensus.cc:2272] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:07:44.458703 2000 ts_tablet_manager.cc:1916] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:44.459301 1707 ts_tablet_manager.cc:1929] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:44.459391 1707 log.cc:1199] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/8925bce2f75b48e4b436dcc35503c9ef
I20260501 02:07:44.459636 1707 ts_tablet_manager.cc:1950] T 8925bce2f75b48e4b436dcc35503c9ef P 77ec954487e04de09106f2bef8dda8e4: Deleting consensus metadata
I20260501 02:07:44.460232 1496 catalog_manager.cc:5002] TS 77ec954487e04de09106f2bef8dda8e4 (127.18.96.193:39369): tablet 8925bce2f75b48e4b436dcc35503c9ef (table test-table [id=e9f05fe847ea499e9a5aab35180e1798]) successfully deleted
I20260501 02:07:44.460382 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1548
I20260501 02:07:44.460978 2000 ts_tablet_manager.cc:1929] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:44.461055 2000 log.cc:1199] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/8925bce2f75b48e4b436dcc35503c9ef
I20260501 02:07:44.461373 2000 ts_tablet_manager.cc:1950] T 8925bce2f75b48e4b436dcc35503c9ef P 23a6d60c57234b08973a1a8c0c64047d: Deleting consensus metadata
I20260501 02:07:44.462141 1999 ts_tablet_manager.cc:1916] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:07:44.462955 1495 catalog_manager.cc:5002] TS 23a6d60c57234b08973a1a8c0c64047d (127.18.96.194:42939): tablet 8925bce2f75b48e4b436dcc35503c9ef (table test-table [id=e9f05fe847ea499e9a5aab35180e1798]) successfully deleted
I20260501 02:07:44.464224 1999 ts_tablet_manager.cc:1929] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:07:44.464300 1999 log.cc:1199] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/8925bce2f75b48e4b436dcc35503c9ef
I20260501 02:07:44.464601 1999 ts_tablet_manager.cc:1950] T 8925bce2f75b48e4b436dcc35503c9ef P 999365ef0dbe4b7ca7593af3b198f22a: Deleting consensus metadata
I20260501 02:07:44.465512 1494 catalog_manager.cc:5002] TS 999365ef0dbe4b7ca7593af3b198f22a (127.18.96.195:35139): tablet 8925bce2f75b48e4b436dcc35503c9ef (table test-table [id=e9f05fe847ea499e9a5aab35180e1798]) successfully deleted
I20260501 02:07:44.467705 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1702
I20260501 02:07:44.473598 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1839
I20260501 02:07:44.479420 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 1477
I20260501 02:07:44.485759 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--webserver_interface=127.18.96.254
--webserver_port=43137
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:44.581677 2001 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:44.581874 2001 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:44.581961 2001 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:44.585224 2001 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:07:44.585275 2001 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:44.585299 2001 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:44.585356 2001 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:07:44.585388 2001 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:07:44.589380 2001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=43137
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:44.590365 2001 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:44.591178 2001 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:44.596175 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.596181 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.596396 2001 server_base.cc:1061] running on GCE node
W20260501 02:07:44.596194 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.597007 2001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:44.597764 2001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:44.598932 2001 hybrid_clock.cc:648] HybridClock initialized: now 1777601264598923 us; error 29 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:44.601471 2001 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:07:44.602519 2001 webserver.cc:492] Webserver started at http://127.18.96.254:43137/ using document root <none> and password file <none>
I20260501 02:07:44.603026 2001 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:44.603097 2001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:44.606125 2001 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:07:44.607865 2016 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.608731 2001 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:44.608886 2001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "55d1676472604eb8bcddebac42c42215"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "e74ed1f6ecbce5080f995726ab2f2d79"
server_key_iv: "feecc70b5c443617954cc0831bcaef1f"
server_key_version: "encryptionkey@0"
I20260501 02:07:44.609229 2001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:44.620930 2001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:44.623116 2001 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:44.623288 2001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:44.630052 2001 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40969
I20260501 02:07:44.630066 2068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40969 every 8 connection(s)
I20260501 02:07:44.630977 2001 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:07:44.634639 2069 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:07:44.636006 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap starting.
I20260501 02:07:44.640807 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2001
I20260501 02:07:44.641351 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:39369
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=34277
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:07:44.652668 2069 log.cc:826] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Log is configured to *not* fsync() on all Append() calls
I20260501 02:07:44.663048 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:07:44.663404 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215: Bootstrap complete.
I20260501 02:07:44.666827 2069 raft_consensus.cc:359] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:44.667100 2069 raft_consensus.cc:740] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Initialized, Role: FOLLOWER
I20260501 02:07:44.667526 2069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:44.667622 2069 raft_consensus.cc:399] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:07:44.667698 2069 raft_consensus.cc:493] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:07:44.667794 2069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 2 FOLLOWER]: Advancing to term 3
I20260501 02:07:44.669147 2069 raft_consensus.cc:515] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:44.669471 2069 leader_election.cc:304] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55d1676472604eb8bcddebac42c42215; no voters:
I20260501 02:07:44.669790 2069 leader_election.cc:290] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260501 02:07:44.669867 2073 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 3 FOLLOWER]: Leader election won for term 3
I20260501 02:07:44.670073 2073 raft_consensus.cc:697] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [term 3 LEADER]: Becoming Leader. State: Replica: 55d1676472604eb8bcddebac42c42215, State: Running, Role: LEADER
I20260501 02:07:44.670329 2073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } }
I20260501 02:07:44.670855 2069 sys_catalog.cc:565] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:07:44.671137 2074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:44.671152 2073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55d1676472604eb8bcddebac42c42215. Latest consensus state: current_term: 3 leader_uuid: "55d1676472604eb8bcddebac42c42215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55d1676472604eb8bcddebac42c42215" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40969 } } }
I20260501 02:07:44.671361 2073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:44.671270 2074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55d1676472604eb8bcddebac42c42215 [sys.catalog]: This master's current role is: LEADER
I20260501 02:07:44.671710 2077 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:07:44.676262 2077 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:07:44.677052 2077 catalog_manager.cc:1269] Loaded cluster ID: 0108d21bfea14acc82b6063388e2021f
I20260501 02:07:44.677111 2077 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20260501 02:07:44.677943 2077 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f48871f9dcd google::LogMessage::Fail() at ??:0
@ 0x7f48871fdb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f48871f97cc google::LogMessage::Flush() at ??:0
@ 0x7f48871faf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f488c6c167e _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f488c6c2516 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f488c6be9db _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f488c70805d _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561592d67244 std::function<>::operator()() at ??:0
@ 0x7f488870e6e3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f488870f00b _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4888710757 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x561592d67244 std::function<>::operator()() at ??:0
@ 0x7f48886fe7ea kudu::Thread::SuperviseThread() at ??:0
@ 0x7f48890426db start_thread at ??:0
@ 0x7f48863b061f clone at ??:0
W20260501 02:07:44.777994 2071 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:44.778270 2071 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:44.778347 2071 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:44.782915 2071 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:44.782990 2071 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:44.783097 2071 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:07:44.788852 2071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:39369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=34277
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:44.790097 2071 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:44.791110 2071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:44.797801 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.798050 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.798341 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.798318 2071 server_base.cc:1061] running on GCE node
I20260501 02:07:44.798751 2071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:44.799258 2071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:44.800428 2071 hybrid_clock.cc:648] HybridClock initialized: now 1777601264800417 us; error 29 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:44.802995 2071 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:07:44.803961 2071 webserver.cc:492] Webserver started at http://127.18.96.193:34277/ using document root <none> and password file <none>
I20260501 02:07:44.804474 2071 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:44.804527 2071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:44.807765 2071 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:07:44.809477 2120 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.810425 2071 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:07:44.810551 2071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "77ec954487e04de09106f2bef8dda8e4"
format_stamp: "Formatted at 2026-05-01 02:07:42 on dist-test-slave-2j90"
server_key: "69cdfd91847681dc6c20a14dd682d2f7"
server_key_iv: "3164baff08ad191c1f3fbde45798233e"
server_key_version: "encryptionkey@0"
I20260501 02:07:44.810899 2071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:44.825080 2071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:44.827893 2071 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:44.828078 2071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:44.828593 2071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:44.829414 2071 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:44.829484 2071 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.829550 2071 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:44.829584 2071 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.840260 2071 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:39369
I20260501 02:07:44.840353 2233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:39369 every 8 connection(s)
I20260501 02:07:44.841207 2071 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:07:44.847787 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2071
I20260501 02:07:44.848188 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:42939
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=39427
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:44.938870 2236 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:07:44.843202 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:45013)
0501 02:07:44.843651 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.843691 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0501 02:07:44.844615 (+ 924us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:07:44.938129 (+ 93514us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
W20260501 02:07:44.939430 2234 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:40969 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:40969: Client connection negotiation failed: client connection to 127.18.96.254:40969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 02:07:44.940182 2236 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.939889 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:55103)
0501 02:07:44.940044 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.940109 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260501 02:07:44.940946 2236 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:44.940702 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:46749)
0501 02:07:44.940834 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:44.940883 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260501 02:07:44.941130 2234 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:07:44.965941 2237 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:44.966246 2237 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:44.966351 2237 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:44.969673 2237 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:44.969770 2237 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:44.969894 2237 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:07:44.974058 2237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:42939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=39427
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:44.975065 2237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:44.975884 2237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:44.982237 2243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:44.982239 2242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.982326 2237 server_base.cc:1061] running on GCE node
W20260501 02:07:44.982470 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:44.982857 2237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:44.983361 2237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:44.984560 2237 hybrid_clock.cc:648] HybridClock initialized: now 1777601264984545 us; error 57 us; skew 500 ppm
May 01 02:07:44 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601264, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:44.987248 2237 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:07:44.988129 2237 webserver.cc:492] Webserver started at http://127.18.96.194:39427/ using document root <none> and password file <none>
I20260501 02:07:44.988595 2237 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:44.988637 2237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:44.991640 2237 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260501 02:07:44.993295 2252 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:44.994369 2237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260501 02:07:44.994457 2237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "23a6d60c57234b08973a1a8c0c64047d"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "0d9714ee3abf3773be617f524d8f625a"
server_key_iv: "367c1995a88955c19a00ad869c11d77e"
server_key_version: "encryptionkey@0"
I20260501 02:07:44.994797 2237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:45.007933 2237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:45.010583 2237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:45.010723 2237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:45.011159 2237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:45.011900 2237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:45.011945 2237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:45.011981 2237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:45.012001 2237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:45.021142 2237 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42939
I20260501 02:07:45.021188 2365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42939 every 8 connection(s)
I20260501 02:07:45.022245 2237 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:07:45.024845 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2237
I20260501 02:07:45.025163 2368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.023952 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:33079)
0501 02:07:45.024424 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.024498 (+ 74us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":109,"threads_started":1}
I20260501 02:07:45.025313 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:35139
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:40969
--webserver_port=39197
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:07:45.025588 2366 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:40969 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:40969: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
I20260501 02:07:45.026255 2368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.025941 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:37309)
0501 02:07:45.026120 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.026171 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260501 02:07:45.026932 2368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.026681 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:36165)
0501 02:07:45.026832 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.026880 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260501 02:07:45.027088 2366 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:07:45.124791 2369 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:07:45.124995 2369 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:07:45.125100 2369 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:07:45.129657 2369 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:07:45.129746 2369 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:07:45.129906 2369 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:07:45.135370 2369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:35139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=39197
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:07:45.136358 2369 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:07:45.137148 2369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:07:45.143314 2375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:45.143369 2374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:07:45.143369 2377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:07:45.143680 2369 server_base.cc:1061] running on GCE node
I20260501 02:07:45.144047 2369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:07:45.144548 2369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:07:45.145725 2369 hybrid_clock.cc:648] HybridClock initialized: now 1777601265145718 us; error 28 us; skew 500 ppm
May 01 02:07:45 dist-test-slave-2j90 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601265, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:07:45.148365 2369 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:07:45.149410 2369 webserver.cc:492] Webserver started at http://127.18.96.195:39197/ using document root <none> and password file <none>
I20260501 02:07:45.149940 2369 fs_manager.cc:362] Metadata directory not provided
I20260501 02:07:45.150070 2369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:07:45.153332 2369 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260501 02:07:45.155146 2384 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:45.156030 2369 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:07:45.156162 2369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "999365ef0dbe4b7ca7593af3b198f22a"
format_stamp: "Formatted at 2026-05-01 02:07:43 on dist-test-slave-2j90"
server_key: "6b9915c92c159087da7961e8712e4d81"
server_key_iv: "a4334feddddaacf4680cf76652beb669"
server_key_version: "encryptionkey@0"
I20260501 02:07:45.156540 2369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:07:45.184743 2369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:07:45.187215 2369 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:07:45.187418 2369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:07:45.187991 2369 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:07:45.188953 2369 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:07:45.189023 2369 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:45.189100 2369 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:07:45.189152 2369 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:07:45.198598 2369 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:35139
I20260501 02:07:45.198623 2497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:35139 every 8 connection(s)
I20260501 02:07:45.199563 2369 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:07:45.201311 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2369
I20260501 02:07:45.202749 2500 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.201597 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:56737)
0501 02:07:45.202039 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.202125 (+ 86us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
W20260501 02:07:45.203080 2498 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:40969 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:40969: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
I20260501 02:07:45.203641 2500 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.203435 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:33759)
0501 02:07:45.203545 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.203593 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260501 02:07:45.204200 2500 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.203990 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:59039)
0501 02:07:45.204110 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.204151 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260501 02:07:45.204366 2498 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260501 02:07:45.942476 2502 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:45.941678 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:56395)
0501 02:07:45.942021 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:45.942067 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260501 02:07:46.028048 2503 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:46.027596 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:59113)
0501 02:07:46.027913 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:46.027957 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":81,"threads_started":1}
I20260501 02:07:46.205440 2504 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:46.204903 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:60817)
0501 02:07:46.205240 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:46.205299 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":94,"threads_started":1}
I20260501 02:07:46.943630 2505 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:46.943236 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:48529)
0501 02:07:46.943485 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:46.943532 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260501 02:07:47.029074 2506 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:47.028703 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:40807)
0501 02:07:47.028933 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:47.028976 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260501 02:07:47.206804 2507 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:47.206332 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:43429)
0501 02:07:47.206605 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:47.206652 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260501 02:07:47.945019 2508 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:47.944553 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:41237)
0501 02:07:47.944821 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:47.944887 (+ 66us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260501 02:07:48.030256 2509 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:48.029808 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:48879)
0501 02:07:48.030121 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.030168 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260501 02:07:48.208092 2510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:48.207611 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:43875)
0501 02:07:48.207927 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.207978 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":102,"threads_started":1}
I20260501 02:07:48.946287 2511 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:48.945807 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:43069)
0501 02:07:48.946112 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:48.946157 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":117,"threads_started":1}
I20260501 02:07:49.031342 2512 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:49.030941 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:52661)
0501 02:07:49.031197 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.031252 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
I20260501 02:07:49.209270 2513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:49.208865 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:35125)
0501 02:07:49.209127 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.209173 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":72,"threads_started":1}
I20260501 02:07:49.947464 2514 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:49.947071 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:56273)
0501 02:07:49.947320 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:49.947367 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260501 02:07:50.032550 2515 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:50.032124 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:43573)
0501 02:07:50.032417 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:50.032463 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
I20260501 02:07:50.210557 2516 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:50.210176 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:48471)
0501 02:07:50.210399 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:50.210442 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260501 02:07:50.948719 2517 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:50.948305 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:39209)
0501 02:07:50.948564 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:50.948611 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260501 02:07:51.033659 2518 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:51.033255 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:57733)
0501 02:07:51.033503 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:51.033573 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
I20260501 02:07:51.211788 2519 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:51.211362 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:34639)
0501 02:07:51.211639 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:51.211702 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20260501 02:07:51.950012 2520 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:51.949578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:36091)
0501 02:07:51.949848 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:51.949893 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":72,"threads_started":1}
I20260501 02:07:52.034736 2521 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:52.034357 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:56503)
0501 02:07:52.034608 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:52.034650 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260501 02:07:52.212929 2522 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:52.212467 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:54901)
0501 02:07:52.212787 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:52.212832 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":81,"threads_started":1}
I20260501 02:07:52.951071 2523 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:52.950685 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:48917)
0501 02:07:52.950932 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:52.950975 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260501 02:07:53.035872 2524 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:53.035437 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:49715)
0501 02:07:53.035710 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:53.035765 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260501 02:07:53.214041 2525 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:53.213627 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:59997)
0501 02:07:53.213855 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:53.213897 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260501 02:07:53.952339 2526 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:53.951941 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:53671)
0501 02:07:53.952203 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:53.952243 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260501 02:07:54.037093 2527 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:54.036690 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:45945)
0501 02:07:54.036934 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:54.036982 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260501 02:07:54.215243 2528 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:54.214799 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:52485)
0501 02:07:54.215073 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:54.215119 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260501 02:07:54.953446 2529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:54.953036 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:56601)
0501 02:07:54.953303 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:54.953350 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260501 02:07:55.038326 2530 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:55.037779 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:55309)
0501 02:07:55.038142 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:55.038189 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":93,"threads_started":1}
I20260501 02:07:55.216457 2531 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:55.216007 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:36851)
0501 02:07:55.216293 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:55.216344 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":75,"threads_started":1}
I20260501 02:07:55.954607 2532 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:55.954192 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:43705)
0501 02:07:55.954456 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:55.954503 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260501 02:07:56.039475 2533 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:56.039038 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:39367)
0501 02:07:56.039323 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:56.039371 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
I20260501 02:07:56.217703 2534 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:56.217245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:50367)
0501 02:07:56.217495 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:56.217540 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260501 02:07:56.955786 2535 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:56.955393 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:57105)
0501 02:07:56.955645 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:56.955689 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260501 02:07:57.040637 2536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:57.040231 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:57653)
0501 02:07:57.040478 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:57.040548 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260501 02:07:57.218861 2537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:57.218438 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:37927)
0501 02:07:57.218695 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:57.218743 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260501 02:07:57.956897 2538 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:57.956476 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:55373)
0501 02:07:57.956747 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:57.956798 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260501 02:07:58.041844 2539 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:58.041318 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:55587)
0501 02:07:58.041546 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:58.041738 (+ 192us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260501 02:07:58.220041 2540 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:58.219649 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:40495)
0501 02:07:58.219903 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:58.219949 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260501 02:07:58.958127 2541 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:58.957676 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:51405)
0501 02:07:58.957934 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:58.958027 (+ 93us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260501 02:07:59.043058 2542 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:59.042660 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:36729)
0501 02:07:59.042910 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:59.042960 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260501 02:07:59.221138 2543 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:59.220715 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.195:36335)
0501 02:07:59.221002 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:59.221045 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260501 02:07:59.959352 2544 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:07:59.958947 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.193:55611)
0501 02:07:59.959184 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 02:07:59.959230 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
I20260501 02:08:00.044312 2545 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:00.043872 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40969 (local address 127.18.96.194:38585)
0501 02:08:00.044118 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:00.044189 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:40969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260501 02:08:00.206081 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2071
I20260501 02:08:00.211829 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2237
I20260501 02:08:00.217389 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2369
2026-05-01T02:08:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19790 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:08:00 dist-test-slave-2j90 krb5kdc[2548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:08:00 dist-test-slave-2j90 krb5kdc[2548](info): set up 2 sockets
May 01 02:08:00 dist-test-slave-2j90 krb5kdc[2548](info): commencing operation
krb5kdc: starting...
W20260501 02:08:02.202734 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.963s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:08:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:08:02.337262 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:41623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:02.434082 2564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:02.434371 2564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:02.434464 2564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:02.437700 2564 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260501 02:08:02.437786 2564 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260501 02:08:02.437815 2564 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260501 02:08:02.437855 2564 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:08:02.437901 2564 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:02.437964 2564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:02.438009 2564 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:08:02.438042 2564 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:08:02.442304 2564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:41623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:02.443252 2564 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:02.444031 2564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:02.449455 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:02.449476 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:02.449541 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:02.449828 2564 server_base.cc:1061] running on GCE node
I20260501 02:08:02.450254 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:02.451071 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:02.452222 2564 hybrid_clock.cc:648] HybridClock initialized: now 1777601282452198 us; error 39 us; skew 500 ppm
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:02.454689 2564 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:08:02.455628 2564 webserver.cc:492] Webserver started at http://127.18.96.254:36199/ using document root <none> and password file <none>
I20260501 02:08:02.456084 2564 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:02.456126 2564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:02.456269 2564 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:08:02.457747 2564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1f0ff5e9d318efee1d0a985a8e84565b"
server_key_iv: "6f466da6d2ad6d00c3974b8d1c831120"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.458230 2564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1f0ff5e9d318efee1d0a985a8e84565b"
server_key_iv: "6f466da6d2ad6d00c3974b8d1c831120"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.461176 2564 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260501 02:08:02.463124 2579 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:02.463980 2564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:08:02.464133 2564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1f0ff5e9d318efee1d0a985a8e84565b"
server_key_iv: "6f466da6d2ad6d00c3974b8d1c831120"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.464226 2564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:02.480957 2564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:02.483595 2564 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:02.483788 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:02.490715 2564 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:41623
I20260501 02:08:02.490713 2631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:41623 every 8 connection(s)
I20260501 02:08:02.491700 2564 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:08:02.492224 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2564
I20260501 02:08:02.492364 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:08:02.492583 18819 external_mini_cluster.cc:1468] Setting key 3525dfc3f932c5c43720b270a4ae7c71
I20260501 02:08:02.494902 2632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:02.501255 2632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap starting.
I20260501 02:08:02.503893 2632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:02.504763 2632 log.cc:826] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:02.505937 2635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:02.493948 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.18.96.254:41623)
0501 02:08:02.494422 (+ 474us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:02.494432 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:02.494469 (+ 37us) server_negotiation.cc:408] Connection header received
0501 02:08:02.495251 (+ 782us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:02.495274 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:02.495604 (+ 330us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:02.495906 (+ 302us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:02.496712 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.498072 (+ 1360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:02.498733 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.498991 (+ 258us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:02.501208 (+ 2217us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:02.501264 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:02.501277 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:02.501338 (+ 61us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:02.503179 (+ 1841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:02.503656 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:02.503689 (+ 33us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:02.503695 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:02.503827 (+ 132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:02.504219 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:02.504223 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:02.504224 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:02.504594 (+ 370us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:02.504771 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:02.505046 (+ 275us) server_negotiation.cc:300] Negotiation successful
0501 02:08:02.505264 (+ 218us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":95,"threads_started":1}
I20260501 02:08:02.506755 2632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: No bootstrap required, opened a new log
I20260501 02:08:02.509093 2632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:02.509253 2632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:02.509339 2632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Initialized, Role: FOLLOWER
I20260501 02:08:02.509774 2632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:02.509917 2632 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:08:02.509992 2632 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:08:02.510093 2632 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:02.510879 2632 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:02.511201 2632 leader_election.cc:304] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c18c89cdc1f4ec3b343d8bbb6e3a48f; no voters:
I20260501 02:08:02.511466 2632 leader_election.cc:290] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:08:02.511598 2637 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:08:02.511809 2637 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 LEADER]: Becoming Leader. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Running, Role: LEADER
I20260501 02:08:02.512099 2637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:02.512456 2632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:08:02.513705 2638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:02.513844 2638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:02.514092 2639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c18c89cdc1f4ec3b343d8bbb6e3a48f. Latest consensus state: current_term: 1 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:02.514217 2639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:02.514340 2641 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:08:02.517002 2641 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:08:02.521384 2641 catalog_manager.cc:1357] Generated new cluster ID: 1211ca4a0c474a8794629e761ddacfec
I20260501 02:08:02.521446 2641 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:08:02.547230 2641 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:08:02.548060 2641 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:08:02.558969 2641 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Generated new TSK 0
I20260501 02:08:02.559520 2641 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:08:02.611084 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:41623
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:02.724679 2660 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:02.724915 2660 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:02.725010 2660 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:02.728394 2660 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:02.728485 2660 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:02.728591 2660 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:08:02.732578 2660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:02.733565 2660 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:02.734421 2660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:02.740698 2665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:02.740710 2666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:02.740780 2660 server_base.cc:1061] running on GCE node
W20260501 02:08:02.740710 2668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:02.741216 2660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:02.741729 2660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:02.742895 2660 hybrid_clock.cc:648] HybridClock initialized: now 1777601282742861 us; error 50 us; skew 500 ppm
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:02.745362 2660 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:08:02.746415 2660 webserver.cc:492] Webserver started at http://127.18.96.193:44807/ using document root <none> and password file <none>
I20260501 02:08:02.746938 2660 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:02.747010 2660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:02.747215 2660 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:08:02.748725 2660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "446be082882748f6894a58f54096317d"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "304c90ac0cd86e66e26ecfbd3210abfb"
server_key_iv: "1a99c11339215dbaea5a5f45f263a993"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.749152 2660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "446be082882748f6894a58f54096317d"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "304c90ac0cd86e66e26ecfbd3210abfb"
server_key_iv: "1a99c11339215dbaea5a5f45f263a993"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.752257 2660 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:08:02.754164 2675 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:02.754981 2660 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:02.755105 2660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "446be082882748f6894a58f54096317d"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "304c90ac0cd86e66e26ecfbd3210abfb"
server_key_iv: "1a99c11339215dbaea5a5f45f263a993"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.755214 2660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:02.772610 2660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:02.777882 2660 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:02.778090 2660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:02.778704 2660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:02.779510 2660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:02.779582 2660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:02.779650 2660 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:02.779692 2660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:02.788651 2660 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44487
I20260501 02:08:02.788674 2788 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44487 every 8 connection(s)
I20260501 02:08:02.789484 2660 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:08:02.796623 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2660
I20260501 02:08:02.796759 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:08:02.797003 18819 external_mini_cluster.cc:1468] Setting key 1a66ba8626f2444cc844e597183a81d1
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:02.801468 2635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:02.791162 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:52239 (local address 127.18.96.254:41623)
0501 02:08:02.791340 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:02.791344 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:02.791856 (+ 512us) server_negotiation.cc:408] Connection header received
0501 02:08:02.792707 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:02.792710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:02.792771 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:02.792873 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:02.793921 (+ 1048us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.794505 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:02.795121 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.795321 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:02.797872 (+ 2551us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:02.797898 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:02.797900 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:02.797924 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:02.799409 (+ 1485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:02.799982 (+ 573us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:02.799988 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:02.799992 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:02.800053 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:02.800345 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:02.800350 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:02.800351 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:02.800556 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:02.800665 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:02.801059 (+ 394us) server_negotiation.cc:300] Negotiation successful
0501 02:08:02.801210 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:08:02.801750 2791 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:02.791371 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:52239)
0501 02:08:02.791734 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:02.791761 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:02.792502 (+ 741us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:02.793015 (+ 513us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:02.793022 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:02.793342 (+ 320us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:02.793765 (+ 423us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:02.793775 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.794625 (+ 850us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:02.794629 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:02.795007 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:02.795013 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:02.795203 (+ 190us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:02.796163 (+ 960us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:02.796179 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:02.797736 (+ 1557us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:02.799529 (+ 1793us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:02.799537 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:02.799550 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:02.799861 (+ 311us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:02.800155 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:02.800158 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:02.800159 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:02.800235 (+ 76us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:02.800664 (+ 429us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:02.800668 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:02.800833 (+ 165us) client_negotiation.cc:770] Sending connection context
0501 02:08:02.800978 (+ 145us) client_negotiation.cc:241] Negotiation successful
0501 02:08:02.801126 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260501 02:08:02.802834 2789 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:02.803047 2789 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:02.803453 2789 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:02.804805 2596 ts_manager.cc:194] Registered new tserver with Master: 446be082882748f6894a58f54096317d (127.18.96.193:44487)
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:08:02.805948 2596 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:52239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:08:02.845072 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:41623
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:02.942530 2796 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:02.942765 2796 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:02.942853 2796 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:02.946082 2796 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:02.946166 2796 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:02.946273 2796 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:08:02.950413 2796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:02.951388 2796 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:02.952185 2796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:02.958207 2801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:02.958262 2804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:02.958207 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:02.958628 2796 server_base.cc:1061] running on GCE node
I20260501 02:08:02.959018 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:02.959571 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:02.960743 2796 hybrid_clock.cc:648] HybridClock initialized: now 1777601282960720 us; error 40 us; skew 500 ppm
May 01 02:08:02 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:02.963472 2796 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:08:02.964524 2796 webserver.cc:492] Webserver started at http://127.18.96.194:44731/ using document root <none> and password file <none>
I20260501 02:08:02.965059 2796 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:02.965129 2796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:02.965337 2796 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:08:02.966984 2796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "cf93d0f69eb646fc81832d4e55b935a4"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1babc27abe7ff9738b6b17a6aa2d9738"
server_key_iv: "bda0cb82026d098c7d14f6b1be05cd32"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.967474 2796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "cf93d0f69eb646fc81832d4e55b935a4"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1babc27abe7ff9738b6b17a6aa2d9738"
server_key_iv: "bda0cb82026d098c7d14f6b1be05cd32"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.970738 2796 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:08:02.972645 2811 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:02.973487 2796 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:08:02.973613 2796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "cf93d0f69eb646fc81832d4e55b935a4"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1babc27abe7ff9738b6b17a6aa2d9738"
server_key_iv: "bda0cb82026d098c7d14f6b1be05cd32"
server_key_version: "encryptionkey@0"
I20260501 02:08:02.973706 2796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:02.996116 2796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:02.998579 2796 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:02.998868 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:02.999471 2796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:03.000396 2796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:03.000478 2796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:03.000550 2796 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:03.000588 2796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:03.009748 2796 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42453
I20260501 02:08:03.009778 2924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42453 every 8 connection(s)
I20260501 02:08:03.010689 2796 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
May 01 02:08:03 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601282, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:03.019898 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2796
I20260501 02:08:03.020027 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:08:03.020247 18819 external_mini_cluster.cc:1468] Setting key 3181e8509455d359a1413d8c8007bd12
I20260501 02:08:03.022640 2635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.012285 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:42905 (local address 127.18.96.254:41623)
0501 02:08:03.012445 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.012450 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.013034 (+ 584us) server_negotiation.cc:408] Connection header received
0501 02:08:03.013883 (+ 849us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.013886 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.013933 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.014055 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:03.015123 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.015678 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.016246 (+ 568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.016401 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.018873 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:03.018888 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:03.018891 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:03.018914 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:03.020706 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.021185 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.021189 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.021191 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.021244 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.021584 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.021587 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.021589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.021757 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:03.021860 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.022418 (+ 558us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.022514 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:08:03.023411 2927 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.012534 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:42905)
0501 02:08:03.012909 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.012937 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.013673 (+ 736us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.014186 (+ 513us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.014193 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.014545 (+ 352us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:03.015001 (+ 456us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.015011 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.015777 (+ 766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.015780 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.016151 (+ 371us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.016157 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.016293 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.017247 (+ 954us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:03.017263 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:03.018735 (+ 1472us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:03.020819 (+ 2084us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:03.020824 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:03.020834 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:03.021087 (+ 253us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:03.021362 (+ 275us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:03.021365 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:03.021366 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:03.021454 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:03.021883 (+ 429us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:03.021888 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:03.022149 (+ 261us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.022365 (+ 216us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.022609 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260501 02:08:03.024564 2925 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:03.024796 2925 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:03.025233 2925 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:03.026314 2596 ts_manager.cc:194] Registered new tserver with Master: cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194:42453)
I20260501 02:08:03.027086 2596 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:42905
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:08:03.067099 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:41623
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:03.165915 2932 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:03.166182 2932 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:03.166239 2932 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:03.169505 2932 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:03.169564 2932 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:03.169637 2932 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:08:03.173591 2932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:03.174454 2932 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:03.175168 2932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:03.181064 2940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:03.181169 2938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:03.181278 2937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:03.181201 2932 server_base.cc:1061] running on GCE node
I20260501 02:08:03.181630 2932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:03.182149 2932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:03.183322 2932 hybrid_clock.cc:648] HybridClock initialized: now 1777601283183313 us; error 38 us; skew 500 ppm
May 01 02:08:03 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:03.186048 2932 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:08:03.186985 2932 webserver.cc:492] Webserver started at http://127.18.96.195:43341/ using document root <none> and password file <none>
I20260501 02:08:03.187566 2932 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:03.187721 2932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:03.187973 2932 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:08:03.189561 2932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "5c123c83f2924b0b837ec056ef168a3d"
format_stamp: "Formatted at 2026-05-01 02:08:03 on dist-test-slave-2j90"
server_key: "96cde98df5b8955bd5aa3c3c944fbed2"
server_key_iv: "527f78dfa5f855677bb9ce5c21e4435c"
server_key_version: "encryptionkey@0"
I20260501 02:08:03.190114 2932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "5c123c83f2924b0b837ec056ef168a3d"
format_stamp: "Formatted at 2026-05-01 02:08:03 on dist-test-slave-2j90"
server_key: "96cde98df5b8955bd5aa3c3c944fbed2"
server_key_iv: "527f78dfa5f855677bb9ce5c21e4435c"
server_key_version: "encryptionkey@0"
I20260501 02:08:03.193477 2932 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:08:03.195566 2947 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:03.196470 2932 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:08:03.196614 2932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "5c123c83f2924b0b837ec056ef168a3d"
format_stamp: "Formatted at 2026-05-01 02:08:03 on dist-test-slave-2j90"
server_key: "96cde98df5b8955bd5aa3c3c944fbed2"
server_key_iv: "527f78dfa5f855677bb9ce5c21e4435c"
server_key_version: "encryptionkey@0"
I20260501 02:08:03.196717 2932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:03.209110 2932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:03.211417 2932 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:03.211606 2932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:03.212108 2932 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:03.212917 2932 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:03.212989 2932 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:03.213052 2932 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:03.213104 2932 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:03.221937 2932 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:32915
I20260501 02:08:03.222043 3060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:32915 every 8 connection(s)
I20260501 02:08:03.222863 2932 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:08:03 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601283, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:03.232017 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 2932
I20260501 02:08:03.232158 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:08:03.232411 18819 external_mini_cluster.cc:1468] Setting key bce7c3a7df92bf71ff801616be6594f8
I20260501 02:08:03.234591 2635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.224422 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:57675 (local address 127.18.96.254:41623)
0501 02:08:03.224553 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.224556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.225123 (+ 567us) server_negotiation.cc:408] Connection header received
0501 02:08:03.225910 (+ 787us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.225914 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.225993 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.226092 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:03.227157 (+ 1065us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.227709 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.228335 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.228518 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.231013 (+ 2495us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:03.231036 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:03.231039 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:03.231062 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:03.232652 (+ 1590us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.233151 (+ 499us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.233155 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.233157 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.233213 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.233518 (+ 305us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.233521 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.233523 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.233673 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:03.233775 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.234302 (+ 527us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.234415 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:08:03.235198 3063 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.224661 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:57675)
0501 02:08:03.225005 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.225033 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.225728 (+ 695us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.226234 (+ 506us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.226241 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.226599 (+ 358us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:03.227013 (+ 414us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.227023 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.227829 (+ 806us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.227840 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.228216 (+ 376us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.228222 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.228417 (+ 195us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.229454 (+ 1037us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:03.229471 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:03.230868 (+ 1397us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:03.232788 (+ 1920us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:03.232794 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:03.232813 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:03.233038 (+ 225us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:03.233309 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:03.233313 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:03.233315 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:03.233410 (+ 95us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:03.233788 (+ 378us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:03.233795 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:03.234058 (+ 263us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.234262 (+ 204us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.234495 (+ 233us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260501 02:08:03.236135 3061 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:03.236343 3061 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:03.236768 3061 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:03.237686 2596 ts_manager.cc:194] Registered new tserver with Master: 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915)
I20260501 02:08:03.238248 2596 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:57675
I20260501 02:08:03.246096 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:08:03.254401 2635 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.247906 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.18.96.254:41623)
0501 02:08:03.248049 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.248053 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.248223 (+ 170us) server_negotiation.cc:408] Connection header received
0501 02:08:03.248322 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.248324 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.248365 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.248432 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:03.249187 (+ 755us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.249797 (+ 610us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.250535 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.250678 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.251746 (+ 1068us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:03.251759 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:03.251761 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:03.251778 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:03.252916 (+ 1138us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.253255 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.253258 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.253259 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.253316 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:03.253532 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:03.253535 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:03.253536 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:03.253731 (+ 195us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:03.253877 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.254092 (+ 215us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.254183 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:08:03.256853 2596 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:08:03.259320 2596 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:08:03.271237 3074 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.266924 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42453 (local address 127.0.0.1:38774)
0501 02:08:03.267462 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.267498 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.267620 (+ 122us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.268360 (+ 740us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.268365 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.268388 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:03.268655 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.268663 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.269840 (+ 1177us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.269844 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.270774 (+ 930us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.270782 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.270894 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.270937 (+ 43us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.271021 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.271091 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":101,"threads_started":1}
I20260501 02:08:03.271677 3076 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.267579 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.0.0.1:44488)
0501 02:08:03.268593 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.268605 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.268690 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.269111 (+ 421us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.269114 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.269135 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:03.269366 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.269372 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.270519 (+ 1147us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.270523 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.271355 (+ 832us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.271367 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.271470 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.271482 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.271525 (+ 43us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.271572 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":70,"threads_started":1}
I20260501 02:08:03.272089 3075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.267466 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.18.96.195:32915)
0501 02:08:03.267930 (+ 464us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.267936 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.267957 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:08:03.268033 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.268037 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.268186 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.268337 (+ 151us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:03.268837 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.269859 (+ 1022us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.271171 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.271719 (+ 548us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.271820 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.271883 (+ 63us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.271954 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":87,"threads_started":1}
I20260501 02:08:03.272424 3072 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.267073 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32915 (local address 127.0.0.1:39858)
0501 02:08:03.267772 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.267784 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.267852 (+ 68us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.268452 (+ 600us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.268454 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.268495 (+ 41us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:03.268704 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.268714 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.269987 (+ 1273us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.269991 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.270799 (+ 808us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.270806 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.270892 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.270937 (+ 45us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.271021 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.271091 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":98,"threads_started":1}
I20260501 02:08:03.272601 3073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.267023 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.18.96.194:42453)
0501 02:08:03.267621 (+ 598us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.267627 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.267650 (+ 23us) server_negotiation.cc:408] Connection header received
0501 02:08:03.267861 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.267867 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.268050 (+ 183us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.268186 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:03.268779 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.269734 (+ 955us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.271533 (+ 1799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.272076 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.272186 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.272282 (+ 96us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.272423 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":85,"threads_started":1}
I20260501 02:08:03.272773 3077 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.267976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.18.96.193:44487)
0501 02:08:03.268324 (+ 348us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.268330 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.268690 (+ 360us) server_negotiation.cc:408] Connection header received
0501 02:08:03.268806 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.268811 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.268977 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.269103 (+ 126us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:03.269487 (+ 384us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.270390 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.271923 (+ 1533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.272363 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.272455 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.272543 (+ 88us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.272626 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260501 02:08:03.275305 2995 tablet_service.cc:1511] Processing CreateTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (DEFAULT_TABLE table=test-table [id=10d4838a87644a20ad949d9fdf71877a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:03.275339 2859 tablet_service.cc:1511] Processing CreateTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (DEFAULT_TABLE table=test-table [id=10d4838a87644a20ad949d9fdf71877a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:03.275337 2723 tablet_service.cc:1511] Processing CreateTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (DEFAULT_TABLE table=test-table [id=10d4838a87644a20ad949d9fdf71877a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:03.276139 2995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2bc84e387f4c91b47c3bc4b3b71404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:03.276178 2723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2bc84e387f4c91b47c3bc4b3b71404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:03.276178 2859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2bc84e387f4c91b47c3bc4b3b71404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:03.281266 3078 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Bootstrap starting.
I20260501 02:08:03.282372 3079 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Bootstrap starting.
I20260501 02:08:03.283802 3078 tablet_bootstrap.cc:654] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:03.284370 3079 tablet_bootstrap.cc:654] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:03.284672 3078 log.cc:826] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:03.285128 3079 log.cc:826] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:03.285377 3080 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Bootstrap starting.
I20260501 02:08:03.286500 3078 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: No bootstrap required, opened a new log
I20260501 02:08:03.286698 3078 ts_tablet_manager.cc:1403] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260501 02:08:03.286966 3079 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: No bootstrap required, opened a new log
I20260501 02:08:03.287139 3079 ts_tablet_manager.cc:1403] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:08:03.287362 3080 tablet_bootstrap.cc:654] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:03.288111 3080 log.cc:826] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:03.289940 3080 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: No bootstrap required, opened a new log
I20260501 02:08:03.290138 3080 ts_tablet_manager.cc:1403] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:08:03.290254 3078 raft_consensus.cc:359] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.290503 3078 raft_consensus.cc:385] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:03.290586 3078 raft_consensus.cc:740] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c123c83f2924b0b837ec056ef168a3d, State: Initialized, Role: FOLLOWER
I20260501 02:08:03.290452 3079 raft_consensus.cc:359] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.290678 3079 raft_consensus.cc:385] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:03.290776 3079 raft_consensus.cc:740] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf93d0f69eb646fc81832d4e55b935a4, State: Initialized, Role: FOLLOWER
I20260501 02:08:03.291002 3078 consensus_queue.cc:260] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.291188 3079 consensus_queue.cc:260] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.291647 3061 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
I20260501 02:08:03.291829 3078 ts_tablet_manager.cc:1434] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260501 02:08:03.292033 3079 ts_tablet_manager.cc:1434] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260501 02:08:03.292361 2925 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
I20260501 02:08:03.293661 3080 raft_consensus.cc:359] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.293829 3080 raft_consensus.cc:385] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:03.293910 3080 raft_consensus.cc:740] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446be082882748f6894a58f54096317d, State: Initialized, Role: FOLLOWER
I20260501 02:08:03.294447 3080 consensus_queue.cc:260] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.295270 2789 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
I20260501 02:08:03.295475 3080 ts_tablet_manager.cc:1434] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260501 02:08:03.336390 3085 raft_consensus.cc:493] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:08:03.336580 3085 raft_consensus.cc:515] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.337705 3085 leader_election.cc:290] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915), 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:03.340462 2927 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.337856 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32915 (local address 127.18.96.194:59385)
0501 02:08:03.338010 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.338038 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.338143 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.338424 (+ 281us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.338427 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.338448 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:03.338648 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.338653 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.339404 (+ 751us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.339410 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.340027 (+ 617us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.340033 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.340207 (+ 174us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.340224 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.340277 (+ 53us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.340338 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260501 02:08:03.340885 3075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.337993 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:59385 (local address 127.18.96.195:32915)
0501 02:08:03.338158 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.338171 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.338188 (+ 17us) server_negotiation.cc:408] Connection header received
0501 02:08:03.338240 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.338244 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.338307 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.338400 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:03.338770 (+ 370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.339296 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.340180 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.340674 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.340713 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.340762 (+ 49us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.340801 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:08:03.341439 3087 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.337954 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.18.96.194:57677)
0501 02:08:03.338393 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.338403 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:03.338512 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:03.338771 (+ 259us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:03.338775 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:03.338793 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:03.339036 (+ 243us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.339042 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.339754 (+ 712us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.339760 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:03.340475 (+ 715us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:03.340482 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.341145 (+ 663us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.341156 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:08:03.341257 (+ 101us) client_negotiation.cc:241] Negotiation successful
0501 02:08:03.341317 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":173,"threads_started":1}
I20260501 02:08:03.341457 3077 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.338183 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:57677 (local address 127.18.96.193:44487)
0501 02:08:03.338329 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.338333 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.338482 (+ 149us) server_negotiation.cc:408] Connection header received
0501 02:08:03.338631 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.338634 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.338676 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.338761 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:03.339156 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.339640 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.340641 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.341046 (+ 405us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.341085 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.341315 (+ 230us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.341371 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:08:03.341455 3015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a2bc84e387f4c91b47c3bc4b3b71404" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c123c83f2924b0b837ec056ef168a3d" is_pre_election: true
I20260501 02:08:03.341713 3015 raft_consensus.cc:2468] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 0.
I20260501 02:08:03.342191 2743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a2bc84e387f4c91b47c3bc4b3b71404" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446be082882748f6894a58f54096317d" is_pre_election: true
I20260501 02:08:03.342239 2815 leader_election.cc:304] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c123c83f2924b0b837ec056ef168a3d, cf93d0f69eb646fc81832d4e55b935a4; no voters:
I20260501 02:08:03.342432 2743 raft_consensus.cc:2468] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 0.
I20260501 02:08:03.342484 3085 raft_consensus.cc:2804] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:08:03.342535 3085 raft_consensus.cc:493] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:08:03.342588 3085 raft_consensus.cc:3060] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:03.343652 3085 raft_consensus.cc:515] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.344022 3085 leader_election.cc:290] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915), 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:03.344309 3015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a2bc84e387f4c91b47c3bc4b3b71404" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c123c83f2924b0b837ec056ef168a3d"
I20260501 02:08:03.344360 2743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a2bc84e387f4c91b47c3bc4b3b71404" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446be082882748f6894a58f54096317d"
I20260501 02:08:03.344425 3015 raft_consensus.cc:3060] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:03.344453 2743 raft_consensus.cc:3060] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:03.345579 2743 raft_consensus.cc:2468] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 1.
I20260501 02:08:03.345579 3015 raft_consensus.cc:2468] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 1.
I20260501 02:08:03.345855 2812 leader_election.cc:304] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446be082882748f6894a58f54096317d, cf93d0f69eb646fc81832d4e55b935a4; no voters:
I20260501 02:08:03.346035 3085 raft_consensus.cc:2804] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:08:03.346241 3085 raft_consensus.cc:697] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 LEADER]: Becoming Leader. State: Replica: cf93d0f69eb646fc81832d4e55b935a4, State: Running, Role: LEADER
I20260501 02:08:03.346517 3085 consensus_queue.cc:237] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:03.349234 2595 catalog_manager.cc:5671] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "cf93d0f69eb646fc81832d4e55b935a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } health_report { overall_health: UNKNOWN } } }
I20260501 02:08:03.368717 3073 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.366140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.18.96.194:42453)
0501 02:08:03.366261 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:03.366265 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:03.366276 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:08:03.366335 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:03.366337 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:03.366381 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:03.366454 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:08:03.366734 (+ 280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.367198 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:03.367815 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:03.367947 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:03.368024 (+ 77us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:03.368406 (+ 382us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:03.368490 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:03.368596 (+ 106us) server_negotiation.cc:300] Negotiation successful
0501 02:08:03.368631 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:08:03.373648 2743 raft_consensus.cc:1275] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Refusing update from remote peer cf93d0f69eb646fc81832d4e55b935a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:08:03.373649 3015 raft_consensus.cc:1275] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Refusing update from remote peer cf93d0f69eb646fc81832d4e55b935a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:08:03.374357 3085 consensus_queue.cc:1048] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:08:03.374558 3088 consensus_queue.cc:1048] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:08:03.381419 3092 mvcc.cc:204] Tried to move back new op lower bound from 7281054856692342784 to 7281054856589713408. Current Snapshot: MvccSnapshot[applied={T|T < 7281054856692342784}]
I20260501 02:08:03.381567 3093 mvcc.cc:204] Tried to move back new op lower bound from 7281054856692342784 to 7281054856589713408. Current Snapshot: MvccSnapshot[applied={T|T < 7281054856692342784}]
I20260501 02:08:03.388537 2595 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:08:03.388707 2595 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:08:03.391034 2595 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Sending DeleteTablet for 3 replicas of tablet 8a2bc84e387f4c91b47c3bc4b3b71404
I20260501 02:08:03.391731 2995 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:39858
I20260501 02:08:03.391772 2859 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:38774
I20260501 02:08:03.391921 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2660
I20260501 02:08:03.391899 2723 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:44488
I20260501 02:08:03.392292 3102 tablet_replica.cc:333] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: stopping tablet replica
I20260501 02:08:03.392602 3102 raft_consensus.cc:2243] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:08:03.393014 3102 raft_consensus.cc:2272] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:08:03.393618 3101 tablet_replica.cc:333] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: stopping tablet replica
I20260501 02:08:03.393867 3101 raft_consensus.cc:2243] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:08:03.394176 3101 raft_consensus.cc:2272] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:08:03.394477 3102 ts_tablet_manager.cc:1916] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:08:03.395424 3101 ts_tablet_manager.cc:1916] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:08:03.397840 3101 ts_tablet_manager.cc:1929] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:08:03.398001 3101 log.cc:1199] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/8a2bc84e387f4c91b47c3bc4b3b71404
I20260501 02:08:03.398132 3102 ts_tablet_manager.cc:1929] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:08:03.398276 3102 log.cc:1199] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/8a2bc84e387f4c91b47c3bc4b3b71404
I20260501 02:08:03.398419 3101 ts_tablet_manager.cc:1950] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 5c123c83f2924b0b837ec056ef168a3d: Deleting consensus metadata
I20260501 02:08:03.398684 3102 ts_tablet_manager.cc:1950] T 8a2bc84e387f4c91b47c3bc4b3b71404 P cf93d0f69eb646fc81832d4e55b935a4: Deleting consensus metadata
I20260501 02:08:03.399755 2580 catalog_manager.cc:5002] TS cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194:42453): tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (table test-table [id=10d4838a87644a20ad949d9fdf71877a]) successfully deleted
I20260501 02:08:03.399755 2583 catalog_manager.cc:5002] TS 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915): tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (table test-table [id=10d4838a87644a20ad949d9fdf71877a]) successfully deleted
W20260501 02:08:03.401078 2580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:44487 (error 108)
I20260501 02:08:03.401191 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2796
I20260501 02:08:03.406697 3076 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:03.406441 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.0.0.1:44492)
0501 02:08:03.406570 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:03.406629 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260501 02:08:03.406873 2580 catalog_manager.cc:4729] TS 446be082882748f6894a58f54096317d (127.18.96.193:44487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a2bc84e387f4c91b47c3bc4b3b71404: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44487: connect: Connection refused (error 111)
I20260501 02:08:03.407451 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2932
I20260501 02:08:03.413297 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 2564
I20260501 02:08:23.420192 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--webserver_interface=127.18.96.254
--webserver_port=36199
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:41623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:23.517346 3104 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:23.517606 3104 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:23.517699 3104 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:23.520802 3104 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260501 02:08:23.520870 3104 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260501 02:08:23.520900 3104 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260501 02:08:23.520947 3104 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:08:23.520988 3104 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:23.521049 3104 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:23.521086 3104 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:08:23.521121 3104 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:08:23.525257 3104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:41623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=36199
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:23.526235 3104 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:23.527025 3104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:23.536304 3113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:23.536377 3104 server_base.cc:1061] running on GCE node
W20260501 02:08:23.536315 3110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:23.536512 3111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:23.537118 3104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:23.538275 3104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:23.539451 3104 hybrid_clock.cc:648] HybridClock initialized: now 1777601303539453 us; error 44 us; skew 500 ppm
May 01 02:08:23 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:23.542850 3104 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:08:23.543749 3104 webserver.cc:492] Webserver started at http://127.18.96.254:36199/ using document root <none> and password file <none>
I20260501 02:08:23.544353 3104 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:23.544445 3104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:23.548185 3104 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:08:23.549938 3120 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:23.550920 3104 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:23.551050 3104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1f0ff5e9d318efee1d0a985a8e84565b"
server_key_iv: "6f466da6d2ad6d00c3974b8d1c831120"
server_key_version: "encryptionkey@0"
I20260501 02:08:23.551486 3104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:23.573082 3104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:23.581683 3104 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:23.581885 3104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:23.589201 3104 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:41623
I20260501 02:08:23.589210 3172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:41623 every 8 connection(s)
I20260501 02:08:23.590334 3104 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:08:23.594408 3173 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:08:23.595711 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3104
I20260501 02:08:23.596156 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:44487
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=44807
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:08:23.596419 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap starting.
I20260501 02:08:23.612672 3173 log.cc:826] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:23.620639 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:23.620990 3173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap complete.
I20260501 02:08:23.624336 3173 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:23.624636 3173 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Initialized, Role: FOLLOWER
I20260501 02:08:23.625073 3173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:23.625176 3173 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:08:23.625260 3173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:08:23.625345 3173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:08:23.626791 3173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:23.627105 3173 leader_election.cc:304] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c18c89cdc1f4ec3b343d8bbb6e3a48f; no voters:
I20260501 02:08:23.627391 3173 leader_election.cc:290] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260501 02:08:23.627475 3177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Leader election won for term 2
I20260501 02:08:23.627691 3177 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 LEADER]: Becoming Leader. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Running, Role: LEADER
I20260501 02:08:23.627995 3177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:23.628393 3173 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:08:23.628821 3177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c18c89cdc1f4ec3b343d8bbb6e3a48f. Latest consensus state: current_term: 2 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:23.628991 3177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:23.629154 3178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:23.629343 3178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:23.629407 3181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:08:23.633726 3181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:08:23.634519 3181 catalog_manager.cc:1269] Loaded cluster ID: 1211ca4a0c474a8794629e761ddacfec
I20260501 02:08:23.634582 3181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:08:23.636816 3181 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:08:23.637482 3181 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Loaded TSK: 0
I20260501 02:08:23.656440 3181 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Generated new TSK 1
I20260501 02:08:23.656651 3181 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:08:23.698549 3175 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:23.698897 3175 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:23.699038 3175 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:23.702392 3175 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:23.702473 3175 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:23.702600 3175 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:08:23.706715 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:44487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=44807
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:23.707659 3175 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:23.708674 3175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:23.714694 3200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:23.714705 3201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:23.714805 3203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:23.715121 3175 server_base.cc:1061] running on GCE node
I20260501 02:08:23.715471 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:23.715972 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:23.717144 3175 hybrid_clock.cc:648] HybridClock initialized: now 1777601303717119 us; error 41 us; skew 500 ppm
May 01 02:08:23 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:23.719766 3175 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:08:23.720739 3175 webserver.cc:492] Webserver started at http://127.18.96.193:44807/ using document root <none> and password file <none>
I20260501 02:08:23.721246 3175 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:23.721328 3175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:23.724458 3175 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:23.726362 3210 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:23.727339 3175 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:23.727449 3175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "446be082882748f6894a58f54096317d"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "304c90ac0cd86e66e26ecfbd3210abfb"
server_key_iv: "1a99c11339215dbaea5a5f45f263a993"
server_key_version: "encryptionkey@0"
I20260501 02:08:23.727825 3175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:23.762952 3175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:23.765708 3175 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:23.765897 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:23.766530 3175 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:23.767884 3217 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 02:08:23.772068 3175 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 02:08:23.772140 3175 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260501 02:08:23.772221 3175 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 02:08:23.773629 3175 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 02:08:23.773697 3175 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260501 02:08:23.774058 3217 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Bootstrap starting.
I20260501 02:08:23.784961 3175 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44487
I20260501 02:08:23.785171 3324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44487 every 8 connection(s)
I20260501 02:08:23.786049 3175 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:08:23.791960 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3175
I20260501 02:08:23.792373 3217 log.cc:826] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:23.792438 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:42453
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=44731
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
May 01 02:08:23 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:23.800081 3217 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:23.800652 3217 tablet_bootstrap.cc:492] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Bootstrap complete.
I20260501 02:08:23.801894 3327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.787889 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:56391 (local address 127.18.96.254:41623)
0501 02:08:23.788685 (+ 796us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:23.788693 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:23.789002 (+ 309us) server_negotiation.cc:408] Connection header received
0501 02:08:23.790052 (+ 1050us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:23.790061 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:23.790269 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:23.790542 (+ 273us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:23.791598 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.792777 (+ 1179us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.794102 (+ 1325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.794401 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.796419 (+ 2018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:23.796441 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:23.796459 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:23.796488 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:23.798324 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:23.798979 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:23.798988 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:23.798996 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:23.799094 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:23.799476 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:23.799483 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:23.799486 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:23.799859 (+ 373us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:23.799977 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:23.800838 (+ 861us) server_negotiation.cc:300] Negotiation successful
0501 02:08:23.801095 (+ 257us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":103,"threads_started":1}
I20260501 02:08:23.802357 3217 ts_tablet_manager.cc:1403] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.018s
I20260501 02:08:23.803267 3328 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.787868 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:56391)
0501 02:08:23.788870 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.788897 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:23.789649 (+ 752us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:23.790583 (+ 934us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:23.790592 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:23.790998 (+ 406us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:23.791430 (+ 432us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.791441 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.792953 (+ 1512us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.792959 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:23.793578 (+ 619us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.793590 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.793758 (+ 168us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.794440 (+ 682us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:23.794463 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:23.796242 (+ 1779us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:23.798462 (+ 2220us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:23.798487 (+ 25us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:23.798499 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:23.798837 (+ 338us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:23.799226 (+ 389us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:23.799230 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:23.799233 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:23.799352 (+ 119us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:23.800072 (+ 720us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:23.800080 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:23.800326 (+ 246us) client_negotiation.cc:770] Sending connection context
0501 02:08:23.802256 (+ 1930us) client_negotiation.cc:241] Negotiation successful
0501 02:08:23.802490 (+ 234us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":105,"threads_started":1}
I20260501 02:08:23.805704 3325 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:23.805932 3325 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:23.805737 3217 raft_consensus.cc:359] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:23.806135 3217 raft_consensus.cc:740] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446be082882748f6894a58f54096317d, State: Initialized, Role: FOLLOWER
I20260501 02:08:23.806521 3325 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:23.806613 3217 consensus_queue.cc:260] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:23.807374 3217 ts_tablet_manager.cc:1434] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:08:23.809234 3137 ts_manager.cc:194] Registered new tserver with Master: 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:23.812901 3137 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:56391
I20260501 02:08:23.815025 3325 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
I20260501 02:08:23.816957 3332 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.812246 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.0.0.1:56950)
0501 02:08:23.813230 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.813262 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:23.813387 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:23.814601 (+ 1214us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:23.814606 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:23.814628 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:23.814841 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.814847 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.815941 (+ 1094us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.815945 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:23.816594 (+ 649us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.816603 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.816717 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.816754 (+ 37us) client_negotiation.cc:770] Sending connection context
0501 02:08:23.816812 (+ 58us) client_negotiation.cc:241] Negotiation successful
0501 02:08:23.816863 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":107,"threads_started":1}
I20260501 02:08:23.817548 3333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.812845 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.18.96.193:44487)
0501 02:08:23.813718 (+ 873us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:23.813724 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:23.813744 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:08:23.813896 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:23.813901 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:23.814408 (+ 507us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:23.814598 (+ 190us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:23.814976 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.815829 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.816713 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.817195 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.817283 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:23.817338 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:08:23.817410 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":81,"threads_started":1}
I20260501 02:08:23.818342 3259 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a2bc84e387f4c91b47c3bc4b3b71404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:56950
I20260501 02:08:23.819042 3334 tablet_replica.cc:333] stopping tablet replica
I20260501 02:08:23.819285 3334 raft_consensus.cc:2243] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:08:23.819517 3334 raft_consensus.cc:2272] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:08:23.820405 3334 ts_tablet_manager.cc:1916] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:08:23.823426 3334 ts_tablet_manager.cc:1929] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:08:23.823498 3334 log.cc:1199] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal/wals/8a2bc84e387f4c91b47c3bc4b3b71404
I20260501 02:08:23.823781 3334 ts_tablet_manager.cc:1950] T 8a2bc84e387f4c91b47c3bc4b3b71404 P 446be082882748f6894a58f54096317d: Deleting consensus metadata
I20260501 02:08:23.825134 3121 catalog_manager.cc:5002] TS 446be082882748f6894a58f54096317d (127.18.96.193:44487): tablet 8a2bc84e387f4c91b47c3bc4b3b71404 (table test-table [id=10d4838a87644a20ad949d9fdf71877a]) successfully deleted
W20260501 02:08:23.900544 3330 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:23.900851 3330 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:23.900964 3330 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:23.904258 3330 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:23.904361 3330 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:23.904489 3330 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:08:23.908557 3330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:42453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=44731
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:23.909489 3330 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:23.910326 3330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:23.916349 3339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:23.916365 3342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:23.916376 3340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:23.916685 3330 server_base.cc:1061] running on GCE node
I20260501 02:08:23.917076 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:23.917598 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:23.918766 3330 hybrid_clock.cc:648] HybridClock initialized: now 1777601303918757 us; error 24 us; skew 500 ppm
May 01 02:08:23 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:23.921511 3330 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:08:23.922583 3330 webserver.cc:492] Webserver started at http://127.18.96.194:44731/ using document root <none> and password file <none>
I20260501 02:08:23.923113 3330 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:23.923202 3330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:23.926342 3330 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:08:23.927985 3349 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:23.928905 3330 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:23.929029 3330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "cf93d0f69eb646fc81832d4e55b935a4"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1babc27abe7ff9738b6b17a6aa2d9738"
server_key_iv: "bda0cb82026d098c7d14f6b1be05cd32"
server_key_version: "encryptionkey@0"
I20260501 02:08:23.929416 3330 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:23.939250 3330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:23.941823 3330 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:23.942046 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:23.942521 3330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:23.943288 3330 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:23.943367 3330 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:23.943434 3330 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:23.943485 3330 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:23.952569 3330 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42453
I20260501 02:08:23.952597 3462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42453 every 8 connection(s)
I20260501 02:08:23.953498 3330 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
May 01 02:08:23 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:23.961695 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3330
I20260501 02:08:23.962306 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:32915
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=43341
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:08:23.965178 3327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.955037 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:35617 (local address 127.18.96.254:41623)
0501 02:08:23.955187 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:23.955191 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:23.955851 (+ 660us) server_negotiation.cc:408] Connection header received
0501 02:08:23.956685 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:23.956688 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:23.956736 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:23.956814 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:23.957859 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.958409 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.959062 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.959191 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.961646 (+ 2455us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:23.961670 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:23.961674 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:23.961703 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:23.963379 (+ 1676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:23.963826 (+ 447us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:23.963830 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:23.963832 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:23.963881 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:23.964181 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:23.964186 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:23.964188 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:23.964348 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:23.964456 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:23.964895 (+ 439us) server_negotiation.cc:300] Negotiation successful
0501 02:08:23.965014 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:08:23.965816 3465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:23.955317 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:35617)
0501 02:08:23.955733 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:23.955762 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:23.956431 (+ 669us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:23.956944 (+ 513us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:23.956951 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:23.957280 (+ 329us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:23.957737 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.957747 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.958568 (+ 821us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:23.958572 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:23.958963 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:23.958969 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:23.959116 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:23.960071 (+ 955us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:23.960087 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:23.961495 (+ 1408us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:23.963491 (+ 1996us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:23.963496 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:23.963504 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:23.963714 (+ 210us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:23.963972 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:23.963976 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:23.963978 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:23.964059 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:23.964468 (+ 409us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:23.964476 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:23.964666 (+ 190us) client_negotiation.cc:770] Sending connection context
0501 02:08:23.964844 (+ 178us) client_negotiation.cc:241] Negotiation successful
0501 02:08:23.965042 (+ 198us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260501 02:08:23.966899 3463 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:23.967154 3463 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:23.967743 3463 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:23.968842 3137 ts_manager.cc:194] Registered new tserver with Master: cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194:42453)
I20260501 02:08:23.969398 3137 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:35617
W20260501 02:08:24.061311 3466 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:24.061491 3466 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:24.061546 3466 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:24.064783 3466 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:24.064838 3466 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:24.064962 3466 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:08:24.069116 3466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:32915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=43341
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:24.070142 3466 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:24.070937 3466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:24.077119 3472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:24.077169 3474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:24.077291 3466 server_base.cc:1061] running on GCE node
W20260501 02:08:24.077327 3471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:24.077741 3466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:24.078301 3466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:24.079454 3466 hybrid_clock.cc:648] HybridClock initialized: now 1777601304079424 us; error 40 us; skew 500 ppm
May 01 02:08:24 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601304, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:24.082235 3466 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:08:24.083204 3466 webserver.cc:492] Webserver started at http://127.18.96.195:43341/ using document root <none> and password file <none>
I20260501 02:08:24.083740 3466 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:24.083828 3466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:24.086972 3466 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260501 02:08:24.088609 3481 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:24.089430 3466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260501 02:08:24.089562 3466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "5c123c83f2924b0b837ec056ef168a3d"
format_stamp: "Formatted at 2026-05-01 02:08:03 on dist-test-slave-2j90"
server_key: "96cde98df5b8955bd5aa3c3c944fbed2"
server_key_iv: "527f78dfa5f855677bb9ce5c21e4435c"
server_key_version: "encryptionkey@0"
I20260501 02:08:24.089934 3466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:24.119531 3466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:24.122156 3466 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:24.122339 3466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:24.122867 3466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:24.123713 3466 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:24.123780 3466 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:24.123850 3466 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:24.123900 3466 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:24.133687 3466 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:32915
I20260501 02:08:24.133711 3594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:32915 every 8 connection(s)
I20260501 02:08:24.134735 3466 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:08:24.138176 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3466
May 01 02:08:24 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601304, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:08:24.145870 3599 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.138836 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.18.96.254:41623)
0501 02:08:24.139158 (+ 322us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.139163 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.139178 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:08:24.139316 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.139319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.139368 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.139458 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:24.140298 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.140784 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.141484 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.141706 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.142681 (+ 975us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:24.142698 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:24.142701 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:24.142729 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:24.144403 (+ 1674us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.144793 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.144797 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.144799 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.144848 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.145075 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.145078 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.145080 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.145212 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:24.145396 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.145594 (+ 198us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.145697 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":66,"threads_started":1}
I20260501 02:08:24.146756 3327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.136308 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:38959 (local address 127.18.96.254:41623)
0501 02:08:24.136482 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.136488 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.137197 (+ 709us) server_negotiation.cc:408] Connection header received
0501 02:08:24.138017 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.138021 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.138079 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.138175 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:24.139375 (+ 1200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.139895 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.140526 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.140706 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.143461 (+ 2755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:24.143482 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:24.143484 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:24.143514 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:24.144883 (+ 1369us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.145356 (+ 473us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.145360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.145362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.145405 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.145678 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.145681 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.145683 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.145853 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:24.145960 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.146513 (+ 553us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.146639 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:08:24.147480 3597 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.136675 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:38959)
0501 02:08:24.137049 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.137090 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.137812 (+ 722us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.138336 (+ 524us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:24.138351 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:24.138707 (+ 356us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:24.139222 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.139233 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.140025 (+ 792us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.140029 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:24.140407 (+ 378us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.140414 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.140607 (+ 193us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.141688 (+ 1081us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:24.141714 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:24.143321 (+ 1607us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:24.145000 (+ 1679us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:24.145006 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:24.145016 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:24.145234 (+ 218us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:24.145492 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:24.145496 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:24.145498 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:24.145581 (+ 83us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:24.145998 (+ 417us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:24.146006 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:24.146227 (+ 221us) client_negotiation.cc:770] Sending connection context
0501 02:08:24.146466 (+ 239us) client_negotiation.cc:241] Negotiation successful
0501 02:08:24.146680 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260501 02:08:24.148366 3595 heartbeater.cc:344] Connected to a master server at 127.18.96.254:41623
I20260501 02:08:24.148556 3595 heartbeater.cc:461] Registering TS with master...
I20260501 02:08:24.148947 3595 heartbeater.cc:507] Master 127.18.96.254:41623 requested a full tablet report, sending...
I20260501 02:08:24.149832 3137 ts_manager.cc:194] Registered new tserver with Master: 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915)
I20260501 02:08:24.150431 3137 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:38959
I20260501 02:08:24.159123 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:08:24.162096 3327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.159643 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.18.96.254:41623)
0501 02:08:24.159761 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.159764 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.159890 (+ 126us) server_negotiation.cc:408] Connection header received
0501 02:08:24.160015 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.160017 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.160078 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.160143 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:08:24.160459 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.160892 (+ 433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.161498 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.161618 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.161676 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:24.161788 (+ 112us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0501 02:08:24.161947 (+ 159us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38408: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260501 02:08:24.162158 3327 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38408: token expired
W20260501 02:08:24.162459 3066 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.96.254:41623)
I20260501 02:08:24.168570 3327 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.162774 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.18.96.254:41623)
0501 02:08:24.162904 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.162907 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.162919 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:08:24.163026 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.163029 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.163070 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.163147 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:24.163789 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.164236 (+ 447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.164850 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.165045 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.166008 (+ 963us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:24.166023 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:24.166025 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:24.166056 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:24.167151 (+ 1095us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.167528 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.167532 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.167534 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.167582 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:24.167819 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:24.167823 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:24.167825 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:24.167980 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:24.168115 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.168339 (+ 224us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.168437 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:08:24.170441 3137 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:08:24.172822 3137 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:08:24.180019 3259 tablet_service.cc:1511] Processing CreateTablet for tablet 42a42d138b63453da8c2d55c2988c701 (DEFAULT_TABLE table=test-table [id=efca44ade13d4230a1b1b32de4849255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:24.180652 3259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a42d138b63453da8c2d55c2988c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:24.182986 3332 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.178950 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42453 (local address 127.0.0.1:60940)
0501 02:08:24.179049 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.179063 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.179132 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.180117 (+ 985us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:24.180120 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:24.180137 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:24.180391 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.180398 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.181760 (+ 1362us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.181764 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:24.182669 (+ 905us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.182679 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.182764 (+ 85us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.182778 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:08:24.182815 (+ 37us) client_negotiation.cc:241] Negotiation successful
0501 02:08:24.182875 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260501 02:08:24.183154 3605 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.179618 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32915 (local address 127.0.0.1:55820)
0501 02:08:24.179878 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.179898 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.179975 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.180602 (+ 627us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:24.180605 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:24.180623 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:24.180851 (+ 228us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.180858 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.182188 (+ 1330us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.182194 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:24.182754 (+ 560us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.182761 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.182895 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.182909 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:08:24.182960 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:08:24.183010 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":61,"threads_started":1}
I20260501 02:08:24.183295 3217 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Bootstrap starting.
I20260501 02:08:24.184029 3606 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.179793 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.18.96.195:32915)
0501 02:08:24.180137 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.180143 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.180210 (+ 67us) server_negotiation.cc:408] Connection header received
0501 02:08:24.180287 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.180291 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.180476 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.180618 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:24.180965 (+ 347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.182085 (+ 1120us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.182898 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.183696 (+ 798us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.183774 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.183823 (+ 49us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.183875 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":100,"threads_started":1}
I20260501 02:08:24.184213 3217 tablet_bootstrap.cc:654] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:24.184233 3604 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.179115 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60940 (local address 127.18.96.194:42453)
0501 02:08:24.179660 (+ 545us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.179667 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.179688 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:08:24.179783 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.179788 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.179955 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.180143 (+ 188us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:24.180516 (+ 373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.181627 (+ 1111us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.183289 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.183811 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.183904 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.183966 (+ 62us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.184035 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":158,"threads_started":1}
I20260501 02:08:24.185707 3217 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: No bootstrap required, opened a new log
I20260501 02:08:24.185796 3217 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260501 02:08:24.186230 3217 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.186355 3217 raft_consensus.cc:385] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:24.186398 3217 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446be082882748f6894a58f54096317d, State: Initialized, Role: FOLLOWER
I20260501 02:08:24.186522 3217 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.186781 3217 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260501 02:08:24.186745 3529 tablet_service.cc:1511] Processing CreateTablet for tablet 42a42d138b63453da8c2d55c2988c701 (DEFAULT_TABLE table=test-table [id=efca44ade13d4230a1b1b32de4849255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:24.186757 3397 tablet_service.cc:1511] Processing CreateTablet for tablet 42a42d138b63453da8c2d55c2988c701 (DEFAULT_TABLE table=test-table [id=efca44ade13d4230a1b1b32de4849255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:08:24.187661 3397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a42d138b63453da8c2d55c2988c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:24.187666 3529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a42d138b63453da8c2d55c2988c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:24.192925 3608 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Bootstrap starting.
I20260501 02:08:24.192925 3609 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Bootstrap starting.
I20260501 02:08:24.194762 3609 tablet_bootstrap.cc:654] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:24.194764 3608 tablet_bootstrap.cc:654] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Neither blocks nor log segments found. Creating new log.
I20260501 02:08:24.195374 3608 log.cc:826] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:24.195379 3609 log.cc:826] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:24.196844 3608 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: No bootstrap required, opened a new log
I20260501 02:08:24.196848 3609 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: No bootstrap required, opened a new log
I20260501 02:08:24.197029 3608 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:08:24.197039 3609 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:08:24.199875 3609 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.199893 3608 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.200024 3609 raft_consensus.cc:385] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:24.200053 3608 raft_consensus.cc:385] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:08:24.200057 3609 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c123c83f2924b0b837ec056ef168a3d, State: Initialized, Role: FOLLOWER
I20260501 02:08:24.200137 3608 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf93d0f69eb646fc81832d4e55b935a4, State: Initialized, Role: FOLLOWER
I20260501 02:08:24.200465 3609 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.200531 3608 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.201041 3595 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
I20260501 02:08:24.201243 3609 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:08:24.201308 3608 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:08:24.201684 3463 heartbeater.cc:499] Master 127.18.96.254:41623 was elected leader, sending a full tablet report...
W20260501 02:08:24.204531 3464 tablet.cc:2404] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:08:24.206629 3613 raft_consensus.cc:493] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:08:24.206748 3613 raft_consensus.cc:515] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.207677 3613 leader_election.cc:290] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915), 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:24.210717 3465 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.207788 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32915 (local address 127.18.96.194:39779)
0501 02:08:24.207954 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.207969 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.208063 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.208319 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:24.208321 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:24.208340 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:24.208667 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.208675 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.209512 (+ 837us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.209516 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:24.210321 (+ 805us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.210330 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.210447 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.210469 (+ 22us) client_negotiation.cc:770] Sending connection context
0501 02:08:24.210515 (+ 46us) client_negotiation.cc:241] Negotiation successful
0501 02:08:24.210572 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260501 02:08:24.211154 3614 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.207930 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.18.96.194:45957)
0501 02:08:24.208311 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.208324 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.208415 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.208688 (+ 273us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:24.208690 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:24.208703 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:08:24.209038 (+ 335us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.209044 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.209853 (+ 809us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.209856 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:24.210613 (+ 757us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:24.210634 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.210755 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.210769 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:08:24.210957 (+ 188us) client_negotiation.cc:241] Negotiation successful
0501 02:08:24.211027 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":91,"threads_started":1}
I20260501 02:08:24.211180 3606 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.207862 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:39779 (local address 127.18.96.195:32915)
0501 02:08:24.207962 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.207968 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.208035 (+ 67us) server_negotiation.cc:408] Connection header received
0501 02:08:24.208170 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.208174 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.208221 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.208297 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:24.208786 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.209410 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.210469 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.210954 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.210986 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.211032 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.211079 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260501 02:08:24.211498 3333 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.208000 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:45957 (local address 127.18.96.193:44487)
0501 02:08:24.208128 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.208134 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.208404 (+ 270us) server_negotiation.cc:408] Connection header received
0501 02:08:24.208544 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.208547 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.208595 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.208674 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:08:24.209155 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.209746 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.210758 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.211224 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.211264 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.211332 (+ 68us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.211378 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:08:24.212030 3279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446be082882748f6894a58f54096317d" is_pre_election: true
I20260501 02:08:24.212273 3279 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 0.
I20260501 02:08:24.212693 3350 leader_election.cc:304] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446be082882748f6894a58f54096317d, cf93d0f69eb646fc81832d4e55b935a4; no voters:
I20260501 02:08:24.212874 3613 raft_consensus.cc:2804] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:08:24.212954 3613 raft_consensus.cc:493] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:08:24.212997 3613 raft_consensus.cc:3060] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:24.213625 3549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c123c83f2924b0b837ec056ef168a3d" is_pre_election: true
I20260501 02:08:24.213845 3549 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 0.
I20260501 02:08:24.214231 3613 raft_consensus.cc:515] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.214612 3613 leader_election.cc:290] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915), 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:24.214901 3549 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c123c83f2924b0b837ec056ef168a3d"
I20260501 02:08:24.215011 3549 raft_consensus.cc:3060] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:24.215081 3279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "cf93d0f69eb646fc81832d4e55b935a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446be082882748f6894a58f54096317d"
I20260501 02:08:24.215232 3279 raft_consensus.cc:3060] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:08:24.215809 3549 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 1.
I20260501 02:08:24.216068 3353 leader_election.cc:304] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c123c83f2924b0b837ec056ef168a3d, cf93d0f69eb646fc81832d4e55b935a4; no voters:
I20260501 02:08:24.216217 3613 raft_consensus.cc:2804] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:08:24.216354 3279 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf93d0f69eb646fc81832d4e55b935a4 in term 1.
I20260501 02:08:24.216424 3613 raft_consensus.cc:697] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 LEADER]: Becoming Leader. State: Replica: cf93d0f69eb646fc81832d4e55b935a4, State: Running, Role: LEADER
I20260501 02:08:24.216785 3613 consensus_queue.cc:237] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:24.219416 3136 catalog_manager.cc:5671] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "cf93d0f69eb646fc81832d4e55b935a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } health_report { overall_health: UNKNOWN } } }
I20260501 02:08:24.237280 3604 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.234335 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.18.96.194:42453)
0501 02:08:24.234466 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:24.234472 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:24.234486 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:08:24.234694 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:24.234697 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:24.234744 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:24.234854 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:08:24.235193 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.235658 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:24.236298 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:24.236502 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:24.236578 (+ 76us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:08:24.236913 (+ 335us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:08:24.237023 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:24.237110 (+ 87us) server_negotiation.cc:300] Negotiation successful
0501 02:08:24.237151 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:08:24.242372 3549 raft_consensus.cc:1275] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Refusing update from remote peer cf93d0f69eb646fc81832d4e55b935a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:08:24.242369 3279 raft_consensus.cc:1275] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Refusing update from remote peer cf93d0f69eb646fc81832d4e55b935a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:08:24.243121 3613 consensus_queue.cc:1048] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:08:24.243360 3615 consensus_queue.cc:1048] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:08:24.248890 3617 mvcc.cc:204] Tried to move back new op lower bound from 7281054942169481216 to 7281054942074253312. Current Snapshot: MvccSnapshot[applied={T|T < 7281054942169481216}]
I20260501 02:08:24.249178 3619 mvcc.cc:204] Tried to move back new op lower bound from 7281054942169481216 to 7281054942074253312. Current Snapshot: MvccSnapshot[applied={T|T < 7281054942169481216}]
I20260501 02:08:24.250424 3618 mvcc.cc:204] Tried to move back new op lower bound from 7281054942169481216 to 7281054942074253312. Current Snapshot: MvccSnapshot[applied={T|T < 7281054942169481216}]
I20260501 02:08:24.257333 3136 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:08:24.257503 3136 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:08:24.259771 3136 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Sending DeleteTablet for 3 replicas of tablet 42a42d138b63453da8c2d55c2988c701
I20260501 02:08:24.260464 3529 tablet_service.cc:1558] Processing DeleteTablet for tablet 42a42d138b63453da8c2d55c2988c701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:24 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:55820
I20260501 02:08:24.260655 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3175
I20260501 02:08:24.260646 3259 tablet_service.cc:1558] Processing DeleteTablet for tablet 42a42d138b63453da8c2d55c2988c701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:24 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:56950
I20260501 02:08:24.261667 3627 tablet_replica.cc:333] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: stopping tablet replica
I20260501 02:08:24.261638 3397 tablet_service.cc:1558] Processing DeleteTablet for tablet 42a42d138b63453da8c2d55c2988c701 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:08:24 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:60940
I20260501 02:08:24.261915 3627 raft_consensus.cc:2243] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:08:24.262236 3627 raft_consensus.cc:2272] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:08:24.262563 3628 tablet_replica.cc:333] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: stopping tablet replica
I20260501 02:08:24.262892 3628 raft_consensus.cc:2243] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:08:24.263206 3628 raft_consensus.cc:2272] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:08:24.264302 3627 ts_tablet_manager.cc:1916] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:08:24.266809 3627 ts_tablet_manager.cc:1929] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:08:24.266901 3627 log.cc:1199] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/42a42d138b63453da8c2d55c2988c701
I20260501 02:08:24.267236 3627 ts_tablet_manager.cc:1950] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Deleting consensus metadata
W20260501 02:08:24.267944 3121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:44487 (error 108)
I20260501 02:08:24.268399 3124 catalog_manager.cc:5002] TS 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915): tablet 42a42d138b63453da8c2d55c2988c701 (table test-table [id=efca44ade13d4230a1b1b32de4849255]) successfully deleted
W20260501 02:08:24.269359 3605 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:08:24.268736 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.0.0.1:56966)
0501 02:08:24.268868 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.268886 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:24.268966 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:24.269247 (+ 281us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
I20260501 02:08:24.269438 3628 ts_tablet_manager.cc:1916] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:08:24.269523 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3330
W20260501 02:08:24.270143 3121 catalog_manager.cc:4729] TS 446be082882748f6894a58f54096317d (127.18.96.193:44487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42a42d138b63453da8c2d55c2988c701: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44487: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 02:08:24.276738 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3466
I20260501 02:08:24.276959 3605 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:24.276669 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42453 (local address 127.0.0.1:60958)
0501 02:08:24.276788 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:24.276869 (+ 81us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.194:42453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260501 02:08:24.282378 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3104
I20260501 02:08:44.289139 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--webserver_interface=127.18.96.254
--webserver_port=36199
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:41623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:08:44.391165 3630 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:44.391393 3630 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:44.391494 3630 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:44.394829 3630 flags.cc:432] Enabled experimental flag: --authn_token_validity_seconds=20
W20260501 02:08:44.394898 3630 flags.cc:432] Enabled experimental flag: --authz_token_validity_seconds=20
W20260501 02:08:44.394927 3630 flags.cc:432] Enabled experimental flag: --tsk_rotation_seconds=20
W20260501 02:08:44.394963 3630 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:08:44.395004 3630 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:44.395066 3630 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:44.395107 3630 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:08:44.395143 3630 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:08:44.399242 3630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:41623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:41623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=36199
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:44.400189 3630 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:44.400979 3630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:44.406492 3638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.406596 3635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.406490 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:44.406737 3630 server_base.cc:1061] running on GCE node
I20260501 02:08:44.407174 3630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:44.408005 3630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:44.409173 3630 hybrid_clock.cc:648] HybridClock initialized: now 1777601324409146 us; error 45 us; skew 500 ppm
May 01 02:08:44 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:44.411978 3630 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:08:44.412995 3630 webserver.cc:492] Webserver started at http://127.18.96.254:36199/ using document root <none> and password file <none>
I20260501 02:08:44.413534 3630 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:44.413610 3630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:44.416869 3630 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:44.418735 3645 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.419690 3630 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:44.419821 3630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1f0ff5e9d318efee1d0a985a8e84565b"
server_key_iv: "6f466da6d2ad6d00c3974b8d1c831120"
server_key_version: "encryptionkey@0"
I20260501 02:08:44.420185 3630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:44.432948 3630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:44.435320 3630 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:44.435498 3630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:44.442677 3630 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:41623
I20260501 02:08:44.442673 3697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:41623 every 8 connection(s)
I20260501 02:08:44.443663 3630 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:08:44.444459 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3630
I20260501 02:08:44.444892 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:44487
--local_ip_for_outbound_sockets=127.18.96.193
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=44807
--webserver_interface=127.18.96.193
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:08:44.447800 3698 sys_catalog.cc:263] Verifying existing consensus state
I20260501 02:08:44.449419 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap starting.
I20260501 02:08:44.466828 3698 log.cc:826] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:44.477932 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:44.478443 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Bootstrap complete.
I20260501 02:08:44.481877 3698 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:44.482251 3698 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Initialized, Role: FOLLOWER
I20260501 02:08:44.482734 3698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:44.482869 3698 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:08:44.482947 3698 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:08:44.483050 3698 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 2 FOLLOWER]: Advancing to term 3
I20260501 02:08:44.484566 3698 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:44.484891 3698 leader_election.cc:304] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c18c89cdc1f4ec3b343d8bbb6e3a48f; no voters:
I20260501 02:08:44.485186 3698 leader_election.cc:290] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260501 02:08:44.485318 3702 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 3 FOLLOWER]: Leader election won for term 3
I20260501 02:08:44.485581 3702 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [term 3 LEADER]: Becoming Leader. State: Replica: 9c18c89cdc1f4ec3b343d8bbb6e3a48f, State: Running, Role: LEADER
I20260501 02:08:44.485906 3702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } }
I20260501 02:08:44.486299 3698 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:08:44.486752 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c18c89cdc1f4ec3b343d8bbb6e3a48f. Latest consensus state: current_term: 3 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:44.486928 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:44.487088 3703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c18c89cdc1f4ec3b343d8bbb6e3a48f" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 41623 } } }
I20260501 02:08:44.487273 3703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f [sys.catalog]: This master's current role is: LEADER
I20260501 02:08:44.487303 3706 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:08:44.492265 3706 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:08:44.492960 3706 catalog_manager.cc:1269] Loaded cluster ID: 1211ca4a0c474a8794629e761ddacfec
I20260501 02:08:44.493029 3706 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:08:44.495285 3706 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:08:44.496032 3706 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Loaded TSK: 0
I20260501 02:08:44.496106 3706 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9c18c89cdc1f4ec3b343d8bbb6e3a48f: Loaded TSK: 1
F20260501 02:08:44.496629 3706 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f858a84adcd google::LogMessage::Fail() at ??:0
@ 0x7f858a84eb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f858a84a7cc google::LogMessage::Flush() at ??:0
@ 0x7f858a84bf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f858bf3fdbe kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f858bf38530 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f858fd3b68b kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f858fd12019 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f858fd128ab _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f858fd59bc0 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f858fd79468 std::function<>::operator()() at ??:0
@ 0x7f858fd121f2 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f858fd1372c kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f858fd0f9db _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f858fd5905d _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fe85e43244 std::function<>::operator()() at ??:0
@ 0x7f858bd5f6e3 kudu::ThreadPool::DispatchThread() at ??:0
W20260501 02:08:44.559445 3700 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:44.559692 3700 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:44.559774 3700 flags.cc:432] Enabled unsafe flag: --never_fsync=true
@ 0x7f858bd6000b _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
W20260501 02:08:44.564379 3700 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:44.564440 3700 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:44.564554 3700 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
@ 0x7f858bd61757 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fe85e43244 std::function<>::operator()() at ??:0
@ 0x7f858bd4f7ea kudu::Thread::SuperviseThread() at ??:0
I20260501 02:08:44.570250 3700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:44487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=44807
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:44.571532 3700 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:44.572485 3700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
@ 0x7f858c6936db start_thread at ??:0
@ 0x7f8589a0161f clone at ??:0
W20260501 02:08:44.580832 3742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.580858 3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.581141 3744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:44.581758 3700 server_base.cc:1061] running on GCE node
I20260501 02:08:44.582104 3700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:44.582615 3700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:44.583775 3700 hybrid_clock.cc:648] HybridClock initialized: now 1777601324583730 us; error 64 us; skew 500 ppm
May 01 02:08:44 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:44.587621 3700 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:08:44.588894 3700 webserver.cc:492] Webserver started at http://127.18.96.193:44807/ using document root <none> and password file <none>
I20260501 02:08:44.589581 3700 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:44.589643 3700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:44.594043 3700 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:08:44.596415 3754 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.597625 3700 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:44.597723 3700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "446be082882748f6894a58f54096317d"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "304c90ac0cd86e66e26ecfbd3210abfb"
server_key_iv: "1a99c11339215dbaea5a5f45f263a993"
server_key_version: "encryptionkey@0"
I20260501 02:08:44.598207 3700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:44.611586 3700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:44.614485 3700 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:44.614682 3700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:44.615326 3700 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:44.616657 3763 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 02:08:44.619712 3700 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 02:08:44.619782 3700 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s user 0.001s sys 0.000s
I20260501 02:08:44.619840 3700 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 02:08:44.621770 3700 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 02:08:44.621832 3700 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260501 02:08:44.622189 3763 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Bootstrap starting.
I20260501 02:08:44.634670 3700 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44487
I20260501 02:08:44.634837 3870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44487 every 8 connection(s)
I20260501 02:08:44.635964 3700 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:08:44.640022 3763 log.cc:826] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:44.640420 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3700
I20260501 02:08:44.640893 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:42453
--local_ip_for_outbound_sockets=127.18.96.194
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=44731
--webserver_interface=127.18.96.194
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:08:44.647647 3763 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:44.648048 3763 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Bootstrap complete.
I20260501 02:08:44.649483 3763 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.011s
I20260501 02:08:44.652118 3763 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:44.652557 3763 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446be082882748f6894a58f54096317d, State: Initialized, Role: FOLLOWER
I20260501 02:08:44.653092 3763 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:44.654040 3763 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260501 02:08:44.745215 3875 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:44.745450 3875 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:44.745538 3875 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:44.749325 3875 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:44.749403 3875 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:44.749574 3875 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
W20260501 02:08:44.753942 3873 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:08:44.638053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:54399)
0501 02:08:44.638521 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.638561 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:44.639420 (+ 859us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:44.753192 (+113772us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260501 02:08:44.754081 3875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:42453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=44731
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
W20260501 02:08:44.754438 3871 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:41623 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:41623: Client connection negotiation failed: client connection to 127.18.96.254:41623: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260501 02:08:44.755180 3873 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.754902 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:54797)
0501 02:08:44.755049 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.755108 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260501 02:08:44.755283 3875 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:44.755806 3873 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.755600 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:52095)
0501 02:08:44.755711 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.755753 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260501 02:08:44.755959 3871 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260501 02:08:44.756191 3875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:44.761999 3884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.762009 3881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.762125 3882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:44.762168 3875 server_base.cc:1061] running on GCE node
I20260501 02:08:44.762573 3875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:44.763095 3875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:44.764266 3875 hybrid_clock.cc:648] HybridClock initialized: now 1777601324764241 us; error 40 us; skew 500 ppm
May 01 02:08:44 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:44.766983 3875 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:08:44.768169 3875 webserver.cc:492] Webserver started at http://127.18.96.194:44731/ using document root <none> and password file <none>
I20260501 02:08:44.768836 3875 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:44.768901 3875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:44.772061 3875 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:08:44.774008 3891 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.774907 3875 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:44.775039 3875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "cf93d0f69eb646fc81832d4e55b935a4"
format_stamp: "Formatted at 2026-05-01 02:08:02 on dist-test-slave-2j90"
server_key: "1babc27abe7ff9738b6b17a6aa2d9738"
server_key_iv: "bda0cb82026d098c7d14f6b1be05cd32"
server_key_version: "encryptionkey@0"
I20260501 02:08:44.775431 3875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:44.797399 3875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:44.800166 3875 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:44.800374 3875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:44.800912 3875 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:44.802042 3898 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260501 02:08:44.805099 3875 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260501 02:08:44.805172 3875 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260501 02:08:44.805253 3875 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260501 02:08:44.806704 3875 ts_tablet_manager.cc:616] Registered 1 tablets
I20260501 02:08:44.806787 3875 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260501 02:08:44.807101 3898 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Bootstrap starting.
I20260501 02:08:44.812126 3898 log.cc:826] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:44.818699 3898 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:44.818893 3875 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42453
I20260501 02:08:44.818930 4006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42453 every 8 connection(s)
I20260501 02:08:44.819106 3898 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Bootstrap complete.
I20260501 02:08:44.819403 3898 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260501 02:08:44.819828 3875 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:08:44.821895 3898 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:44.822301 3898 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf93d0f69eb646fc81832d4e55b935a4, State: Initialized, Role: FOLLOWER
I20260501 02:08:44.822598 4009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.821478 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:54003)
0501 02:08:44.821882 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.822004 (+ 122us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260501 02:08:44.822733 3898 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
W20260501 02:08:44.823009 4007 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:41623 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:41623: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
I20260501 02:08:44.823571 3898 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:08:44.823711 4009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.823406 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:59397)
0501 02:08:44.823531 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.823570 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260501 02:08:44.823766 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 3875
I20260501 02:08:44.824225 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:32915
--local_ip_for_outbound_sockets=127.18.96.195
--tserver_master_addrs=127.18.96.254:41623
--webserver_port=43341
--webserver_interface=127.18.96.195
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:08:44.824939 4009 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.824763 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:44047)
0501 02:08:44.824859 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.824895 (+ 36us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260501 02:08:44.825133 4007 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260501 02:08:44.923472 4011 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:08:44.923658 4011 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:08:44.923713 4011 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:08:44.926903 4011 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:08:44.926954 4011 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:08:44.927024 4011 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:08:44.931051 4011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:32915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=43341
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:41623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:08:44.931882 4011 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:08:44.932670 4011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:08:44.938732 4016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:44.938836 4011 server_base.cc:1061] running on GCE node
W20260501 02:08:44.938956 4017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:08:44.938979 4019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:08:44.939448 4011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:08:44.939936 4011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:08:44.941141 4011 hybrid_clock.cc:648] HybridClock initialized: now 1777601324941110 us; error 49 us; skew 500 ppm
May 01 02:08:44 dist-test-slave-2j90 krb5kdc[2548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:08:44.943895 4011 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:08:44.944866 4011 webserver.cc:492] Webserver started at http://127.18.96.195:43341/ using document root <none> and password file <none>
I20260501 02:08:44.945353 4011 fs_manager.cc:362] Metadata directory not provided
I20260501 02:08:44.945395 4011 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:08:44.948630 4011 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260501 02:08:44.950347 4026 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.951203 4011 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:08:44.951300 4011 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "5c123c83f2924b0b837ec056ef168a3d"
format_stamp: "Formatted at 2026-05-01 02:08:03 on dist-test-slave-2j90"
server_key: "96cde98df5b8955bd5aa3c3c944fbed2"
server_key_iv: "527f78dfa5f855677bb9ce5c21e4435c"
server_key_version: "encryptionkey@0"
I20260501 02:08:44.951676 4011 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:08:44.966063 4011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:08:44.968279 4011 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:08:44.968416 4011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:08:44.968837 4011 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:08:44.969669 4011 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:08:44.969712 4011 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.969748 4011 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:08:44.969762 4011 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:08:44.978448 4011 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:32915
I20260501 02:08:44.978479 4139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:32915 every 8 connection(s)
I20260501 02:08:44.979390 4011 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:08:44.979668 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4011
I20260501 02:08:44.982283 4143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.981292 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:51563)
0501 02:08:44.981680 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.981747 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
W20260501 02:08:44.982622 4140 heartbeater.cc:646] Failed to heartbeat to 127.18.96.254:41623 (0 consecutive failures): Network error: Failed to ping master at 127.18.96.254:41623: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
I20260501 02:08:44.983170 4143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.982967 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:33879)
0501 02:08:44.983078 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.983122 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260501 02:08:44.983729 4143 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:44.983521 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:60143)
0501 02:08:44.983644 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:44.983686 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260501 02:08:44.983879 4140 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260501 02:08:45.756868 4144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.756486 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:40107)
0501 02:08:45.756731 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.756775 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260501 02:08:45.826241 4145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.825737 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:36387)
0501 02:08:45.826099 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.826146 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260501 02:08:45.834313 4146 raft_consensus.cc:493] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:08:45.834496 4146 raft_consensus.cc:515] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:45.835511 4146 leader_election.cc:290] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194:42453), 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915)
May 01 02:08:45 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.195@KRBTEST.COM
May 01 02:08:45 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.194@KRBTEST.COM
I20260501 02:08:45.847829 4144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.835731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32915 (local address 127.18.96.193:37867)
0501 02:08:45.836038 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.836054 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:45.836168 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:45.837749 (+ 1581us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:45.837753 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:45.838005 (+ 252us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:45.838663 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:45.838679 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.840612 (+ 1933us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.840616 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:45.841082 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:45.841094 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.841207 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.841767 (+ 560us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:45.841780 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:45.843383 (+ 1603us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:45.845713 (+ 2330us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.845718 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:45.845728 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:45.846130 (+ 402us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.846444 (+ 314us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846447 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:45.846449 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:45.846530 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.847049 (+ 519us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:45.847055 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:45.847239 (+ 184us) client_negotiation.cc:770] Sending connection context
0501 02:08:45.847424 (+ 185us) client_negotiation.cc:241] Negotiation successful
0501 02:08:45.847659 (+ 235us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"spinlock_wait_cycles":13568}
I20260501 02:08:45.847997 4148 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.835926 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:37867 (local address 127.18.96.195:32915)
0501 02:08:45.836279 (+ 353us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:45.836293 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:45.836333 (+ 40us) server_negotiation.cc:408] Connection header received
0501 02:08:45.837162 (+ 829us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:45.837170 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:45.837502 (+ 332us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.837784 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:45.838843 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.840488 (+ 1645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.841207 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.841472 (+ 265us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.843551 (+ 2079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.843595 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:45.843605 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:45.843633 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:45.845576 (+ 1943us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846269 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.846274 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:45.846278 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:45.846345 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846630 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.846633 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:45.846635 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:45.846932 (+ 297us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.847068 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:45.847621 (+ 553us) server_negotiation.cc:300] Negotiation successful
0501 02:08:45.847817 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":73,"threads_started":1}
I20260501 02:08:45.848023 4149 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.835926 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:55699 (local address 127.18.96.194:42453)
0501 02:08:45.836285 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:45.836293 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:45.836333 (+ 40us) server_negotiation.cc:408] Connection header received
0501 02:08:45.836674 (+ 341us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:45.836680 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:45.837056 (+ 376us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:45.837388 (+ 332us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:45.838842 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.840140 (+ 1298us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.840830 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.841028 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.843862 (+ 2834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:45.843880 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:45.843891 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:45.843918 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:45.845672 (+ 1754us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846266 (+ 594us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.846306 (+ 40us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:45.846310 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:45.846375 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846643 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:45.846647 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:45.846648 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:45.846944 (+ 296us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:45.847080 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:45.847599 (+ 519us) server_negotiation.cc:300] Negotiation successful
0501 02:08:45.847813 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":84,"threads_started":1}
I20260501 02:08:45.848456 4147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.835731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42453 (local address 127.18.96.193:55699)
0501 02:08:45.836038 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.836054 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:45.836171 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:45.837417 (+ 1246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:45.837421 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:45.838005 (+ 584us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:45.838663 (+ 658us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:45.838679 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.840278 (+ 1599us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:45.840281 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:45.840719 (+ 438us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:45.840725 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:45.840884 (+ 159us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:45.841571 (+ 687us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:45.841603 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:45.843706 (+ 2103us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:45.845821 (+ 2115us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.845824 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:45.845827 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:45.846136 (+ 309us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.846481 (+ 345us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:45.846484 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:45.846486 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:45.846552 (+ 66us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:45.847068 (+ 516us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:45.847071 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:45.847241 (+ 170us) client_negotiation.cc:770] Sending connection context
0501 02:08:45.847424 (+ 183us) client_negotiation.cc:241] Negotiation successful
0501 02:08:45.847758 (+ 334us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":164,"spinlock_wait_cycles":27392,"thread_start_us":102,"threads_started":1}
I20260501 02:08:45.849936 3961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "446be082882748f6894a58f54096317d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf93d0f69eb646fc81832d4e55b935a4" is_pre_election: true
I20260501 02:08:45.850268 3961 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446be082882748f6894a58f54096317d in term 1.
W20260501 02:08:45.850292 3760 leader_election.cc:343] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915): Not found: Tablet not found: 42a42d138b63453da8c2d55c2988c701
I20260501 02:08:45.850943 3757 leader_election.cc:304] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 446be082882748f6894a58f54096317d, cf93d0f69eb646fc81832d4e55b935a4; no voters: 5c123c83f2924b0b837ec056ef168a3d
I20260501 02:08:45.851156 4146 raft_consensus.cc:2804] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260501 02:08:45.851222 4146 raft_consensus.cc:493] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:08:45.851262 4146 raft_consensus.cc:3060] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 1 FOLLOWER]: Advancing to term 2
I20260501 02:08:45.852846 4146 raft_consensus.cc:515] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:45.853204 4146 leader_election.cc:290] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 election: Requested vote from peers cf93d0f69eb646fc81832d4e55b935a4 (127.18.96.194:42453), 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915)
I20260501 02:08:45.853538 3961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42a42d138b63453da8c2d55c2988c701" candidate_uuid: "446be082882748f6894a58f54096317d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf93d0f69eb646fc81832d4e55b935a4"
I20260501 02:08:45.853668 3961 raft_consensus.cc:3060] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 1 FOLLOWER]: Advancing to term 2
W20260501 02:08:45.853945 3760 leader_election.cc:343] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915): Not found: Tablet not found: 42a42d138b63453da8c2d55c2988c701
I20260501 02:08:45.855391 3961 raft_consensus.cc:2468] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446be082882748f6894a58f54096317d in term 2.
I20260501 02:08:45.855746 3757 leader_election.cc:304] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 446be082882748f6894a58f54096317d, cf93d0f69eb646fc81832d4e55b935a4; no voters: 5c123c83f2924b0b837ec056ef168a3d
I20260501 02:08:45.855922 4146 raft_consensus.cc:2804] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 2 FOLLOWER]: Leader election won for term 2
I20260501 02:08:45.856145 4146 raft_consensus.cc:697] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [term 2 LEADER]: Becoming Leader. State: Replica: 446be082882748f6894a58f54096317d, State: Running, Role: LEADER
I20260501 02:08:45.856462 4146 consensus_queue.cc:237] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:45.856937 4147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.856702 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:40519)
0501 02:08:45.856814 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.856860 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260501 02:08:45.984817 4152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:45.984462 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:44339)
0501 02:08:45.984696 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:45.984738 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260501 02:08:46.308799 3961 raft_consensus.cc:1275] T 42a42d138b63453da8c2d55c2988c701 P cf93d0f69eb646fc81832d4e55b935a4 [term 2 FOLLOWER]: Refusing update from remote peer 446be082882748f6894a58f54096317d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260501 02:08:46.309614 4146 consensus_queue.cc:1048] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260501 02:08:46.311674 4145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.311431 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:41625)
0501 02:08:46.311552 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.311604 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260501 02:08:46.314239 4147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.313910 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:57249)
0501 02:08:46.314069 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.314141 (+ 72us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260501 02:08:46.314594 3760 consensus_peers.cc:597] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d -> Peer 5c123c83f2924b0b837ec056ef168a3d (127.18.96.195:32915): Couldn't send request to peer 5c123c83f2924b0b837ec056ef168a3d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42a42d138b63453da8c2d55c2988c701. This is attempt 1: this message will repeat every 5th retry.
I20260501 02:08:46.315259 4145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.314993 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:56413)
0501 02:08:46.315165 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.315205 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260501 02:08:46.878034 4158 ts_tablet_manager.cc:933] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Initiating tablet copy from peer 446be082882748f6894a58f54096317d (127.18.96.193:44487)
I20260501 02:08:46.878594 4158 tablet_copy_client.cc:323] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.96.193:44487
May 01 02:08:46 dist-test-slave-2j90 krb5kdc[2548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601324, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.193@KRBTEST.COM
I20260501 02:08:46.887173 4160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.879121 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44487 (local address 127.18.96.195:40913)
0501 02:08:46.879479 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.879496 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:08:46.879612 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:08:46.879988 (+ 376us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:08:46.879992 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:08:46.880416 (+ 424us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:08:46.880676 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:46.880682 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.881517 (+ 835us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.881520 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:08:46.881869 (+ 349us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:08:46.881874 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.882006 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.882517 (+ 511us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:08:46.882530 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:08:46.883899 (+ 1369us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:08:46.885874 (+ 1975us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:46.885879 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:46.885882 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:46.886124 (+ 242us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:46.886367 (+ 243us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:08:46.886370 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:08:46.886371 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:08:46.886412 (+ 41us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:08:46.886720 (+ 308us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:08:46.886724 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:08:46.886827 (+ 103us) client_negotiation.cc:770] Sending connection context
0501 02:08:46.886940 (+ 113us) client_negotiation.cc:241] Negotiation successful
0501 02:08:46.887038 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":149,"threads_started":1}
I20260501 02:08:46.887259 4161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.879291 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:40913 (local address 127.18.96.193:44487)
0501 02:08:46.879560 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0501 02:08:46.879566 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:08:46.879592 (+ 26us) server_negotiation.cc:408] Connection header received
0501 02:08:46.879704 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:08:46.879709 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:08:46.879851 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:08:46.879953 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:08:46.880783 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.881404 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:08:46.881998 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:08:46.882162 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:08:46.884032 (+ 1870us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:08:46.884055 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:08:46.884065 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:08:46.884090 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:08:46.885780 (+ 1690us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:46.886238 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:46.886242 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:46.886244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:46.886289 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:08:46.886465 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:08:46.886467 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:08:46.886468 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:08:46.886635 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:08:46.886723 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0501 02:08:46.886990 (+ 267us) server_negotiation.cc:300] Negotiation successful
0501 02:08:46.887097 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260501 02:08:46.888195 3845 tablet_copy_service.cc:140] P 446be082882748f6894a58f54096317d: Received BeginTabletCopySession request for tablet 42a42d138b63453da8c2d55c2988c701 from peer 5c123c83f2924b0b837ec056ef168a3d ({username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:40913)
I20260501 02:08:46.888368 3845 tablet_copy_service.cc:161] P 446be082882748f6894a58f54096317d: Beginning new tablet copy session on tablet 42a42d138b63453da8c2d55c2988c701 from peer 5c123c83f2924b0b837ec056ef168a3d at {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:40913: session id = 5c123c83f2924b0b837ec056ef168a3d-42a42d138b63453da8c2d55c2988c701
I20260501 02:08:46.889415 3845 tablet_copy_source_session.cc:215] T 42a42d138b63453da8c2d55c2988c701 P 446be082882748f6894a58f54096317d: Tablet Copy: opened 0 blocks and 1 log segments
I20260501 02:08:46.893074 4158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a42d138b63453da8c2d55c2988c701. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:08:46.898609 4158 tablet_copy_client.cc:806] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: tablet copy: Starting download of 0 data blocks...
I20260501 02:08:46.898852 4158 tablet_copy_client.cc:670] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: tablet copy: Starting download of 1 WAL segments...
I20260501 02:08:46.900759 4158 tablet_copy_client.cc:538] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260501 02:08:46.902388 4158 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Bootstrap starting.
I20260501 02:08:46.906915 4158 log.cc:826] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:08:46.912344 4158 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260501 02:08:46.912565 4158 tablet_bootstrap.cc:492] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Bootstrap complete.
I20260501 02:08:46.912786 4158 ts_tablet_manager.cc:1403] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260501 02:08:46.915014 4158 raft_consensus.cc:359] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:46.915187 4158 raft_consensus.cc:740] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c123c83f2924b0b837ec056ef168a3d, State: Initialized, Role: FOLLOWER
I20260501 02:08:46.915558 4158 consensus_queue.cc:260] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf93d0f69eb646fc81832d4e55b935a4" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42453 } } peers { permanent_uuid: "5c123c83f2924b0b837ec056ef168a3d" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32915 } } peers { permanent_uuid: "446be082882748f6894a58f54096317d" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44487 } }
I20260501 02:08:46.916257 4158 ts_tablet_manager.cc:1434] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260501 02:08:46.916719 4160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.916441 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:60279)
0501 02:08:46.916573 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.916637 (+ 64us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260501 02:08:46.917009 3845 tablet_copy_service.cc:342] P 446be082882748f6894a58f54096317d: Request end of tablet copy session 5c123c83f2924b0b837ec056ef168a3d-42a42d138b63453da8c2d55c2988c701 received from {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:40913
I20260501 02:08:46.917150 3845 tablet_copy_service.cc:434] P 446be082882748f6894a58f54096317d: ending tablet copy session 5c123c83f2924b0b837ec056ef168a3d-42a42d138b63453da8c2d55c2988c701 on tablet 42a42d138b63453da8c2d55c2988c701 with peer 5c123c83f2924b0b837ec056ef168a3d
I20260501 02:08:46.917443 4160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:46.917239 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:44125)
0501 02:08:46.917347 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:46.917387 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260501 02:08:47.315232 4164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:47.314868 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:33683)
0501 02:08:47.315101 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.315143 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260501 02:08:47.316461 4165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:47.316007 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:43869)
0501 02:08:47.316339 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.316382 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":179,"threads_started":1}
I20260501 02:08:47.376706 4164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:47.376463 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:56449)
0501 02:08:47.376584 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.376627 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260501 02:08:47.376849 4094 raft_consensus.cc:1217] T 42a42d138b63453da8c2d55c2988c701 P 5c123c83f2924b0b837ec056ef168a3d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260501 02:08:47.377729 4160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:47.377476 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:55795)
0501 02:08:47.377593 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:47.377645 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260501 02:08:48.317670 4168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:48.317208 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:41451)
0501 02:08:48.317516 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.317573 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":148,"threads_started":1}
I20260501 02:08:48.378086 4169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:48.377491 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:39199)
0501 02:08:48.377867 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.377917 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":86,"threads_started":1}
I20260501 02:08:48.378713 4170 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:48.378372 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:32893)
0501 02:08:48.378593 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:48.378637 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":59,"threads_started":1}
I20260501 02:08:49.318745 4171 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:49.318353 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:54585)
0501 02:08:49.318591 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.318635 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260501 02:08:49.379215 4172 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:49.378767 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:34495)
0501 02:08:49.379028 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.379082 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260501 02:08:49.379837 4173 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:49.379402 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:48943)
0501 02:08:49.379668 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:49.379738 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260501 02:08:50.320043 4174 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:50.319620 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:42343)
0501 02:08:50.319904 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.319951 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":100,"threads_started":1}
I20260501 02:08:50.380587 4175 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:50.380094 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:50687)
0501 02:08:50.380388 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.380447 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":69,"threads_started":1}
I20260501 02:08:50.381100 4176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:50.380738 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:44071)
0501 02:08:50.380980 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.381022 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260501 02:08:50.381283 4175 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:50.381053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:36307)
0501 02:08:50.381172 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:50.381217 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260501 02:08:51.321065 4177 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:51.320690 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:53397)
0501 02:08:51.320927 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.320971 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260501 02:08:51.382159 4178 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:51.381731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:35267)
0501 02:08:51.382016 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.382066 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260501 02:08:51.382452 4179 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:51.381948 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:34747)
0501 02:08:51.382247 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:51.382334 (+ 87us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260501 02:08:52.322336 4180 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:52.321852 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:42299)
0501 02:08:52.322167 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.322223 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":126,"threads_started":1}
I20260501 02:08:52.383292 4181 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:52.382871 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:46033)
0501 02:08:52.383143 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.383189 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
I20260501 02:08:52.383548 4182 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:52.383134 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:54881)
0501 02:08:52.383383 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:52.383441 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260501 02:08:53.323436 4183 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:53.323057 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:40937)
0501 02:08:53.323288 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.323332 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260501 02:08:53.384486 4184 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:53.384023 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:42711)
0501 02:08:53.384309 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.384361 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260501 02:08:53.384727 4185 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:53.384322 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:55589)
0501 02:08:53.384559 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:53.384605 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
I20260501 02:08:54.324692 4186 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:54.324269 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:40849)
0501 02:08:54.324541 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.324597 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260501 02:08:54.385870 4187 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:54.385424 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:36545)
0501 02:08:54.385705 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.385759 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260501 02:08:54.385919 4188 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:54.385424 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:59095)
0501 02:08:54.385771 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:54.385820 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":100,"threads_started":1}
I20260501 02:08:55.326086 4189 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:55.325653 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:45733)
0501 02:08:55.325917 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.325990 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260501 02:08:55.387331 4190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:55.386846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:53635)
0501 02:08:55.387168 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.387229 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260501 02:08:55.387331 4191 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:55.386895 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:51487)
0501 02:08:55.387196 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:55.387245 (+ 49us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
I20260501 02:08:56.327167 4192 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:56.326759 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:57719)
0501 02:08:56.327029 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.327074 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260501 02:08:56.388448 4193 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:56.388058 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:57605)
0501 02:08:56.388307 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.388352 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260501 02:08:56.388509 4194 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:56.388114 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:46781)
0501 02:08:56.388378 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:56.388422 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260501 02:08:57.328389 4195 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:57.327950 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:42617)
0501 02:08:57.328202 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.328244 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260501 02:08:57.389645 4196 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:57.389207 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:45397)
0501 02:08:57.389489 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.389546 (+ 57us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260501 02:08:57.389781 4197 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:57.389253 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:53583)
0501 02:08:57.389582 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:57.389650 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260501 02:08:58.329617 4198 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:58.329160 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:52041)
0501 02:08:58.329455 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.329517 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
I20260501 02:08:58.391007 4199 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:58.390550 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:42907)
0501 02:08:58.390861 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.390913 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":90,"threads_started":1}
I20260501 02:08:58.391026 4200 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:58.390564 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:51413)
0501 02:08:58.390896 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:58.390943 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260501 02:08:59.330813 4201 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:59.330435 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.194:39047)
0501 02:08:59.330677 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.330721 (+ 44us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260501 02:08:59.392239 4202 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:59.391838 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.195:56413)
0501 02:08:59.392097 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.392143 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260501 02:08:59.392288 4203 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:08:59.391838 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:41623 (local address 127.18.96.193:47017)
0501 02:08:59.392161 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0501 02:08:59.392203 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.254:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":87,"threads_started":1}
I20260501 02:08:59.986490 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3700
I20260501 02:08:59.992901 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 3875
I20260501 02:08:59.998716 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4011
2026-05-01T02:08:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59782 ms)
[----------] 31 tests from SecurityITest (403831 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:00 dist-test-slave-2j90 krb5kdc[4206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:00 dist-test-slave-2j90 krb5kdc[4206](info): set up 2 sockets
May 01 02:09:00 dist-test-slave-2j90 krb5kdc[4206](info): commencing operation
krb5kdc: starting...
W20260501 02:09:01.986265 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.964s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:02.125222 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38635
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:02.225553 4222 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:02.225787 4222 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:02.225840 4222 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:02.229113 4222 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:02.229171 4222 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:02.229194 4222 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:02.229212 4222 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:02.229228 4222 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:02.233464 4222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:02.234506 4222 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:02.235242 4222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:02.240808 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.240831 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:02.240947 4222 server_base.cc:1061] running on GCE node
W20260501 02:09:02.240836 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:02.241483 4222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:02.242404 4222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:02.243553 4222 hybrid_clock.cc:648] HybridClock initialized: now 1777601342243529 us; error 38 us; skew 500 ppm
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:02.246250 4222 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:02.247279 4222 webserver.cc:492] Webserver started at http://127.18.96.254:37249/ using document root <none> and password file <none>
I20260501 02:09:02.247838 4222 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:02.247912 4222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:02.248121 4222 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:02.249790 4222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "82403159aedd4cdcaa41b4efe1eba4ca"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "1199255f9be79d5868631893cffcedd5"
server_key_iv: "948c0b321e92c074f6cc4c55246c3654"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.250319 4222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "82403159aedd4cdcaa41b4efe1eba4ca"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "1199255f9be79d5868631893cffcedd5"
server_key_iv: "948c0b321e92c074f6cc4c55246c3654"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.253346 4222 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:09:02.255350 4237 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.256285 4222 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:02.256439 4222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "82403159aedd4cdcaa41b4efe1eba4ca"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "1199255f9be79d5868631893cffcedd5"
server_key_iv: "948c0b321e92c074f6cc4c55246c3654"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.256541 4222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:02.276208 4222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:02.279330 4222 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:02.279537 4222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:02.286859 4222 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38635
I20260501 02:09:02.286852 4289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38635 every 8 connection(s)
I20260501 02:09:02.287834 4222 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:02.290503 4290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:02.290508 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4222
I20260501 02:09:02.290716 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:02.290997 18819 external_mini_cluster.cc:1468] Setting key 3bb30f75b1cdb772424932b9e5d6c7ff
I20260501 02:09:02.297247 4290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: Bootstrap starting.
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:02.299331 4290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:02.300185 4290 log.cc:826] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:02.302337 4290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: No bootstrap required, opened a new log
I20260501 02:09:02.304502 4293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.292095 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.18.96.254:38635)
0501 02:09:02.292433 (+ 338us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:02.292445 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:02.292482 (+ 37us) server_negotiation.cc:408] Connection header received
0501 02:09:02.293637 (+ 1155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:02.293662 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:02.294006 (+ 344us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.294335 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:02.295168 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.296370 (+ 1202us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.297101 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.297360 (+ 259us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.299608 (+ 2248us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.299631 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:02.299645 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:02.299676 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:02.301877 (+ 2201us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.302368 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.302377 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.302384 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.302465 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.302705 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.302709 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.302710 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.303077 (+ 367us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.303252 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:02.303609 (+ 357us) server_negotiation.cc:300] Negotiation successful
0501 02:09:02.303819 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":100,"threads_started":1}
I20260501 02:09:02.305327 4290 raft_consensus.cc:359] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } }
I20260501 02:09:02.305531 4290 raft_consensus.cc:385] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:02.305608 4290 raft_consensus.cc:740] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82403159aedd4cdcaa41b4efe1eba4ca, State: Initialized, Role: FOLLOWER
I20260501 02:09:02.306095 4290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } }
I20260501 02:09:02.306209 4290 raft_consensus.cc:399] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:02.306284 4290 raft_consensus.cc:493] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:02.306385 4290 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:02.307212 4290 raft_consensus.cc:515] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } }
I20260501 02:09:02.307497 4290 leader_election.cc:304] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82403159aedd4cdcaa41b4efe1eba4ca; no voters:
I20260501 02:09:02.307781 4290 leader_election.cc:290] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:02.308141 4296 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:02.308370 4296 raft_consensus.cc:697] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [term 1 LEADER]: Becoming Leader. State: Replica: 82403159aedd4cdcaa41b4efe1eba4ca, State: Running, Role: LEADER
I20260501 02:09:02.308650 4296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } }
I20260501 02:09:02.308848 4290 sys_catalog.cc:565] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:02.310266 4297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82403159aedd4cdcaa41b4efe1eba4ca. Latest consensus state: current_term: 1 leader_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } } }
I20260501 02:09:02.310286 4295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82403159aedd4cdcaa41b4efe1eba4ca" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38635 } } }
I20260501 02:09:02.310372 4297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:02.310376 4295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:02.310865 4304 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:02.313778 4304 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:02.318760 4304 catalog_manager.cc:1357] Generated new cluster ID: 3df2fb8918d14f9a966132fcd57727c1
I20260501 02:09:02.318822 4304 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:02.348183 4304 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:02.349316 4304 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:02.359282 4304 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: Generated new TSK 0
I20260501 02:09:02.359941 4304 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:02.408510 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38635
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:02.507027 4318 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:02.507210 4318 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:02.507301 4318 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:02.510701 4318 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:02.510754 4318 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:02.510876 4318 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:02.514984 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:02.516058 4318 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:02.516872 4318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:02.523032 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.523065 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.523169 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:02.523324 4318 server_base.cc:1061] running on GCE node
I20260501 02:09:02.523612 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:02.524087 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:02.525275 4318 hybrid_clock.cc:648] HybridClock initialized: now 1777601342525247 us; error 51 us; skew 500 ppm
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:02.527957 4318 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:09:02.528861 4318 webserver.cc:492] Webserver started at http://127.18.96.193:38777/ using document root <none> and password file <none>
I20260501 02:09:02.529338 4318 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:02.529381 4318 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:02.529521 4318 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:02.531144 4318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "7696efcd193945de9eeb2f2139fc0e1e"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "f5ecbdbe6790cc42318436c0321c56ec"
server_key_iv: "73ae50bfbcbcdc8787b26a4fa0efdaf7"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.531570 4318 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "7696efcd193945de9eeb2f2139fc0e1e"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "f5ecbdbe6790cc42318436c0321c56ec"
server_key_iv: "73ae50bfbcbcdc8787b26a4fa0efdaf7"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.534658 4318 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:09:02.536547 4333 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.537427 4318 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:09:02.537513 4318 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "7696efcd193945de9eeb2f2139fc0e1e"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "f5ecbdbe6790cc42318436c0321c56ec"
server_key_iv: "73ae50bfbcbcdc8787b26a4fa0efdaf7"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.537587 4318 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:02.555286 4318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:02.558012 4318 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:02.558153 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:02.558627 4318 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:02.559396 4318 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:02.559441 4318 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.559479 4318 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:02.559494 4318 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.568536 4318 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:35921
I20260501 02:09:02.568565 4446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:35921 every 8 connection(s)
I20260501 02:09:02.569415 4318 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:09:02.574100 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4318
I20260501 02:09:02.574239 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:02.574496 18819 external_mini_cluster.cc:1468] Setting key dfc697944dbae6681bae1cea18367cc6
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:02.581002 4293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.571040 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39925 (local address 127.18.96.254:38635)
0501 02:09:02.571191 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:02.571196 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:02.571848 (+ 652us) server_negotiation.cc:408] Connection header received
0501 02:09:02.572598 (+ 750us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:02.572601 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:02.572649 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.572723 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:02.574355 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.575009 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.575664 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.575887 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.577863 (+ 1976us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.577887 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:02.577889 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:02.577911 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:02.579239 (+ 1328us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.579703 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.579707 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.579709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.579756 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.580036 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.580040 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.580042 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.580176 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.580280 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:02.580719 (+ 439us) server_negotiation.cc:300] Negotiation successful
0501 02:09:02.580850 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260501 02:09:02.581503 4449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.571340 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38635 (local address 127.18.96.193:39925)
0501 02:09:02.571718 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.571748 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:02.572426 (+ 678us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.572855 (+ 429us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:02.572862 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:02.573169 (+ 307us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:02.574200 (+ 1031us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:02.574220 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.575141 (+ 921us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.575144 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:02.575544 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:02.575551 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.575750 (+ 199us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.576316 (+ 566us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:02.576351 (+ 35us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:02.577724 (+ 1373us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.579357 (+ 1633us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:02.579364 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:02.579373 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:02.579600 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:02.579860 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:02.579863 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:02.579864 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:02.579946 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:02.580281 (+ 335us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.580287 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:02.580475 (+ 188us) client_negotiation.cc:770] Sending connection context
0501 02:09:02.580683 (+ 208us) client_negotiation.cc:241] Negotiation successful
0501 02:09:02.580890 (+ 207us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260501 02:09:02.582792 4447 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38635
I20260501 02:09:02.583002 4447 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
I20260501 02:09:02.583490 4447 heartbeater.cc:507] Master 127.18.96.254:38635 requested a full tablet report, sending...
I20260501 02:09:02.584838 4254 ts_manager.cc:194] Registered new tserver with Master: 7696efcd193945de9eeb2f2139fc0e1e (127.18.96.193:35921)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
I20260501 02:09:02.586282 4254 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:39925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:02.621052 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38635
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:02.717464 4454 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:02.717684 4454 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:02.717768 4454 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:02.722445 4454 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:02.722528 4454 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:02.722632 4454 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:02.726996 4454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:02.727977 4454 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:02.728757 4454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:02.734650 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.734721 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.734725 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:02.734975 4454 server_base.cc:1061] running on GCE node
I20260501 02:09:02.735360 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:02.735797 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:02.736941 4454 hybrid_clock.cc:648] HybridClock initialized: now 1777601342736922 us; error 30 us; skew 500 ppm
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:02.739401 4454 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:09:02.740281 4454 webserver.cc:492] Webserver started at http://127.18.96.194:39559/ using document root <none> and password file <none>
I20260501 02:09:02.740808 4454 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:02.740878 4454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:02.741109 4454 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:02.742826 4454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "e6ca163d5fdd4227a4d0f79e843f13d8"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "9baead43db5d92ff8b864747d4193f78"
server_key_iv: "d78cc963f28ee88fbebd47a670e9803b"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.743299 4454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "e6ca163d5fdd4227a4d0f79e843f13d8"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "9baead43db5d92ff8b864747d4193f78"
server_key_iv: "d78cc963f28ee88fbebd47a670e9803b"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.746439 4454 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:09:02.748548 4469 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.749394 4454 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:02.749538 4454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "e6ca163d5fdd4227a4d0f79e843f13d8"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "9baead43db5d92ff8b864747d4193f78"
server_key_iv: "d78cc963f28ee88fbebd47a670e9803b"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.749643 4454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:02.766661 4454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:02.769589 4454 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:02.769780 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:02.770440 4454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:02.771263 4454 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:02.771345 4454 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.771416 4454 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:02.771449 4454 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.780632 4454 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:36665
I20260501 02:09:02.780664 4582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:36665 every 8 connection(s)
I20260501 02:09:02.781569 4454 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:02.786212 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4454
I20260501 02:09:02.786355 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:02.786586 18819 external_mini_cluster.cc:1468] Setting key b1848769f177b8d5a1ac6d6dfe331552
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:02.793897 4293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.783232 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:53899 (local address 127.18.96.254:38635)
0501 02:09:02.783374 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:02.783378 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:02.784019 (+ 641us) server_negotiation.cc:408] Connection header received
0501 02:09:02.784854 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:02.784858 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:02.784919 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:02.785025 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:02.786688 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.787299 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.788070 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.788204 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.790655 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:02.790682 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:02.790684 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:02.790707 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:02.792099 (+ 1392us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.792591 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.792601 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.792602 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.792644 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:02.792931 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:02.792934 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:02.792936 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:02.793070 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:02.793173 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:02.793631 (+ 458us) server_negotiation.cc:300] Negotiation successful
0501 02:09:02.793733 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:09:02.794441 4585 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.783494 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38635 (local address 127.18.96.194:53899)
0501 02:09:02.783885 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.783915 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:02.784662 (+ 747us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:02.785174 (+ 512us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:02.785181 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:02.785520 (+ 339us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:02.786537 (+ 1017us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:02.786550 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.787437 (+ 887us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:02.787440 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:02.787947 (+ 507us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:02.787957 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:02.788147 (+ 190us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:02.788766 (+ 619us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:02.788793 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:02.790509 (+ 1716us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:02.792229 (+ 1720us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:02.792235 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:02.792245 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:02.792479 (+ 234us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:02.792748 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:02.792751 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:02.792753 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:02.792834 (+ 81us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:02.793183 (+ 349us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:02.793190 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:02.793378 (+ 188us) client_negotiation.cc:770] Sending connection context
0501 02:09:02.793587 (+ 209us) client_negotiation.cc:241] Negotiation successful
0501 02:09:02.793771 (+ 184us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260501 02:09:02.795547 4583 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38635
I20260501 02:09:02.795800 4583 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:09:02.796254 4583 heartbeater.cc:507] Master 127.18.96.254:38635 requested a full tablet report, sending...
I20260501 02:09:02.797277 4254 ts_manager.cc:194] Registered new tserver with Master: e6ca163d5fdd4227a4d0f79e843f13d8 (127.18.96.194:36665)
I20260501 02:09:02.797923 4254 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:53899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:02.832289 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:38635
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:02.928992 4590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:02.929157 4590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:02.929206 4590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:02.932930 4590 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:02.932999 4590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:02.933076 4590 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:02.937410 4590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:38635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:02.938355 4590 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:02.939059 4590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:02.945178 4598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.945160 4596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:02.945160 4595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:02.945533 4590 server_base.cc:1061] running on GCE node
I20260501 02:09:02.945858 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:02.946512 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:02.947705 4590 hybrid_clock.cc:648] HybridClock initialized: now 1777601342947695 us; error 35 us; skew 500 ppm
May 01 02:09:02 dist-test-slave-2j90 krb5kdc[4206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:02.950243 4590 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:09:02.951138 4590 webserver.cc:492] Webserver started at http://127.18.96.195:39041/ using document root <none> and password file <none>
I20260501 02:09:02.951610 4590 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:02.951655 4590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:02.951795 4590 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:02.953364 4590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "088d19fbc5b84d4eb7962f49561d5fc0"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "2fe5604d4d7a70f66e9dda2139c13308"
server_key_iv: "eeaae052dee130429d6e06f4f078f445"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.953820 4590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "088d19fbc5b84d4eb7962f49561d5fc0"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "2fe5604d4d7a70f66e9dda2139c13308"
server_key_iv: "eeaae052dee130429d6e06f4f078f445"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.957082 4590 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:09:02.959203 4605 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.960074 4590 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:02.960176 4590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "088d19fbc5b84d4eb7962f49561d5fc0"
format_stamp: "Formatted at 2026-05-01 02:09:02 on dist-test-slave-2j90"
server_key: "2fe5604d4d7a70f66e9dda2139c13308"
server_key_iv: "eeaae052dee130429d6e06f4f078f445"
server_key_version: "encryptionkey@0"
I20260501 02:09:02.960314 4590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:02.980719 4590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:02.985532 4590 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:02.985739 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:02.986354 4590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:02.987255 4590 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:02.987303 4590 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.987378 4590 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:02.987416 4590 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:02.996222 4590 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:42899
I20260501 02:09:02.996281 4718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:42899 every 8 connection(s)
I20260501 02:09:02.997191 4590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
May 01 02:09:03 dist-test-slave-2j90 krb5kdc[4206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601342, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:03.007467 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4590
I20260501 02:09:03.007632 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:03.007941 18819 external_mini_cluster.cc:1468] Setting key 05cf4a6767505adc44b7f00b13eb1922
I20260501 02:09:03.009047 4293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.998870 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:47819 (local address 127.18.96.254:38635)
0501 02:09:02.999007 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:02.999010 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:02.999593 (+ 583us) server_negotiation.cc:408] Connection header received
0501 02:09:03.000434 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.000438 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.000498 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.000597 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:03.002092 (+ 1495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.002609 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.003243 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.003438 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.005445 (+ 2007us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.005462 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:03.005464 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:03.005490 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:03.006986 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.007549 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.007559 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:03.007560 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:03.007636 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.007976 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.007979 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:03.007980 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:03.008149 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.008225 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.008779 (+ 554us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.008891 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:09:03.009716 4721 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:02.999099 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38635 (local address 127.18.96.195:47819)
0501 02:09:02.999471 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:02.999500 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.000240 (+ 740us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.000768 (+ 528us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.000776 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.001108 (+ 332us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:03.001902 (+ 794us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.001915 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.002731 (+ 816us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.002735 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.003126 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.003132 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.003331 (+ 199us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.003863 (+ 532us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:03.003880 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:03.005288 (+ 1408us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:03.007096 (+ 1808us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:03.007101 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:03.007109 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:03.007434 (+ 325us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:03.007752 (+ 318us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:03.007757 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:03.007759 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:03.007878 (+ 119us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:03.008251 (+ 373us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:03.008258 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:03.008498 (+ 240us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.008722 (+ 224us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.008926 (+ 204us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
I20260501 02:09:03.010674 4719 heartbeater.cc:344] Connected to a master server at 127.18.96.254:38635
I20260501 02:09:03.010893 4719 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:03.011315 4719 heartbeater.cc:507] Master 127.18.96.254:38635 requested a full tablet report, sending...
I20260501 02:09:03.012238 4254 ts_manager.cc:194] Registered new tserver with Master: 088d19fbc5b84d4eb7962f49561d5fc0 (127.18.96.195:42899)
I20260501 02:09:03.012773 4254 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:47819
I20260501 02:09:03.021761 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:03.029440 4293 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.023567 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.18.96.254:38635)
0501 02:09:03.023707 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.023711 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.023873 (+ 162us) server_negotiation.cc:408] Connection header received
0501 02:09:03.023962 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.023965 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.024003 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.024067 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:03.024815 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.025264 (+ 449us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.025849 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.026026 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.026967 (+ 941us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.026979 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:03.026981 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:03.026997 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:03.028225 (+ 1228us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.028549 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.028551 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:03.028552 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:03.028597 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:03.028771 (+ 174us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:03.028773 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:03.028774 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:03.028905 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.028998 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.029163 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.029248 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:09:03.031692 4254 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:03.033875 4254 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:03.045574 4730 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.041474 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:42899 (local address 127.0.0.1:55836)
0501 02:09:03.041773 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.041808 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.042334 (+ 526us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.043017 (+ 683us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.043025 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.043047 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:03.043313 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.043320 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.044403 (+ 1083us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.044406 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.045117 (+ 711us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.045125 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.045255 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.045298 (+ 43us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.045384 (+ 86us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.045448 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260501 02:09:03.046200 4731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.041551 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.18.96.195:42899)
0501 02:09:03.041904 (+ 353us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.041910 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.041932 (+ 22us) server_negotiation.cc:408] Connection header received
0501 02:09:03.042524 (+ 592us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.042530 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.042707 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.042822 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:03.043434 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.044283 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.045281 (+ 998us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.045785 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.045890 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.045984 (+ 94us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.046070 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260501 02:09:03.046427 4733 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.042132 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:36665 (local address 127.0.0.1:56046)
0501 02:09:03.043010 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.043022 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.043110 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.043775 (+ 665us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.043778 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.043795 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:03.044006 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.044011 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.045191 (+ 1180us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.045195 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.046054 (+ 859us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.046070 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.046198 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.046213 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.046263 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.046310 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":106,"threads_started":1}
I20260501 02:09:03.046768 4732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.042078 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35921 (local address 127.0.0.1:58350)
0501 02:09:03.042317 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.042331 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.042420 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.043193 (+ 773us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.043199 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.043215 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:03.043478 (+ 263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.043488 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.044772 (+ 1284us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.044776 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.046436 (+ 1660us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.046446 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.046549 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.046563 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.046611 (+ 48us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.046657 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260501 02:09:03.047127 4735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.042553 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.18.96.194:36665)
0501 02:09:03.042823 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.042830 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.043101 (+ 271us) server_negotiation.cc:408] Connection header received
0501 02:09:03.043320 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.043326 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.043493 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.043632 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:03.044133 (+ 501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.045077 (+ 944us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.046209 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.046733 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.046850 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.046918 (+ 68us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.046995 (+ 77us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260501 02:09:03.048454 4734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.042228 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.18.96.193:35921)
0501 02:09:03.042562 (+ 334us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.042567 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.042586 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:09:03.042654 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.042658 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.042784 (+ 126us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.042912 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:03.043618 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.044657 (+ 1039us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.047038 (+ 2381us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.048109 (+ 1071us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.048192 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.048251 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.048324 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260501 02:09:03.049088 4653 tablet_service.cc:1511] Processing CreateTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 (DEFAULT_TABLE table=test-table [id=257643b4980a46319acf459e41937a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:03.049445 4517 tablet_service.cc:1511] Processing CreateTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 (DEFAULT_TABLE table=test-table [id=257643b4980a46319acf459e41937a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:03.050087 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38b3b4d1db40cf89d662dea4d97730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:03.050356 4517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38b3b4d1db40cf89d662dea4d97730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:03.050834 4381 tablet_service.cc:1511] Processing CreateTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 (DEFAULT_TABLE table=test-table [id=257643b4980a46319acf459e41937a38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:03.051623 4381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38b3b4d1db40cf89d662dea4d97730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:03.055184 4736 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Bootstrap starting.
I20260501 02:09:03.057155 4738 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Bootstrap starting.
I20260501 02:09:03.057713 4736 tablet_bootstrap.cc:654] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:03.058291 4737 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Bootstrap starting.
I20260501 02:09:03.058784 4736 log.cc:826] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:03.059144 4738 tablet_bootstrap.cc:654] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:03.059929 4738 log.cc:826] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:03.060230 4737 tablet_bootstrap.cc:654] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:03.060750 4736 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: No bootstrap required, opened a new log
I20260501 02:09:03.060949 4736 ts_tablet_manager.cc:1403] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260501 02:09:03.060938 4737 log.cc:826] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:03.061790 4738 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: No bootstrap required, opened a new log
I20260501 02:09:03.061987 4738 ts_tablet_manager.cc:1403] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260501 02:09:03.062680 4737 tablet_bootstrap.cc:492] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: No bootstrap required, opened a new log
I20260501 02:09:03.062847 4737 ts_tablet_manager.cc:1403] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:03.064574 4736 raft_consensus.cc:359] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.064808 4736 raft_consensus.cc:385] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:03.064893 4736 raft_consensus.cc:740] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088d19fbc5b84d4eb7962f49561d5fc0, State: Initialized, Role: FOLLOWER
I20260501 02:09:03.065306 4736 consensus_queue.cc:260] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.065610 4738 raft_consensus.cc:359] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.065845 4738 raft_consensus.cc:385] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:03.065932 4738 raft_consensus.cc:740] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7696efcd193945de9eeb2f2139fc0e1e, State: Initialized, Role: FOLLOWER
I20260501 02:09:03.065914 4719 heartbeater.cc:499] Master 127.18.96.254:38635 was elected leader, sending a full tablet report...
I20260501 02:09:03.066143 4736 ts_tablet_manager.cc:1434] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260501 02:09:03.066391 4738 consensus_queue.cc:260] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.066367 4737 raft_consensus.cc:359] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.066538 4737 raft_consensus.cc:385] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:03.066640 4737 raft_consensus.cc:740] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ca163d5fdd4227a4d0f79e843f13d8, State: Initialized, Role: FOLLOWER
I20260501 02:09:03.066991 4447 heartbeater.cc:499] Master 127.18.96.254:38635 was elected leader, sending a full tablet report...
I20260501 02:09:03.067178 4738 ts_tablet_manager.cc:1434] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:09:03.067481 4737 consensus_queue.cc:260] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.068084 4583 heartbeater.cc:499] Master 127.18.96.254:38635 was elected leader, sending a full tablet report...
I20260501 02:09:03.068185 4737 ts_tablet_manager.cc:1434] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260501 02:09:03.070485 4448 tablet.cc:2404] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:03.248113 4720 tablet.cc:2404] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:03.282635 4584 tablet.cc:2404] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:03.306192 4744 raft_consensus.cc:493] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:03.306428 4744 raft_consensus.cc:515] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.307960 4744 leader_election.cc:290] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 088d19fbc5b84d4eb7962f49561d5fc0 (127.18.96.195:42899), 7696efcd193945de9eeb2f2139fc0e1e (127.18.96.193:35921)
I20260501 02:09:03.311404 4746 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.308190 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:42899 (local address 127.18.96.194:46527)
0501 02:09:03.308741 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.308753 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.308873 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.309153 (+ 280us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.309156 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.309175 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:03.309510 (+ 335us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.309516 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.310252 (+ 736us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.310256 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.310954 (+ 698us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.310961 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.311107 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.311125 (+ 18us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.311180 (+ 55us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.311233 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":314,"thread_start_us":92,"threads_started":1}
I20260501 02:09:03.311468 4745 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.308190 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35921 (local address 127.18.96.194:60443)
0501 02:09:03.308556 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.308570 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.308720 (+ 150us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.309022 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.309025 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.309063 (+ 38us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:03.309330 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.309336 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.310272 (+ 936us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.310277 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.311003 (+ 726us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.311012 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.311149 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.311166 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.311225 (+ 59us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.311280 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260501 02:09:03.311862 4731 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.308317 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46527 (local address 127.18.96.195:42899)
0501 02:09:03.308472 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.308476 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.308836 (+ 360us) server_negotiation.cc:408] Connection header received
0501 02:09:03.308984 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.308987 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.309037 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.309140 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:03.309637 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.310138 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.311068 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.311571 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.311611 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.311665 (+ 54us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.311753 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260501 02:09:03.311945 4734 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.308348 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:60443 (local address 127.18.96.193:35921)
0501 02:09:03.308493 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.308497 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.308661 (+ 164us) server_negotiation.cc:408] Connection header received
0501 02:09:03.308844 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.308848 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.308904 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.308996 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:03.309459 (+ 463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.310155 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.311129 (+ 974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.311640 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.311680 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.311741 (+ 61us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.311801 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:09:03.312579 4401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a38b3b4d1db40cf89d662dea4d97730" candidate_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7696efcd193945de9eeb2f2139fc0e1e" is_pre_election: true
I20260501 02:09:03.312839 4673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a38b3b4d1db40cf89d662dea4d97730" candidate_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" is_pre_election: true
I20260501 02:09:03.312870 4401 raft_consensus.cc:2468] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ca163d5fdd4227a4d0f79e843f13d8 in term 0.
I20260501 02:09:03.313069 4673 raft_consensus.cc:2468] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ca163d5fdd4227a4d0f79e843f13d8 in term 0.
I20260501 02:09:03.313361 4473 leader_election.cc:304] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7696efcd193945de9eeb2f2139fc0e1e, e6ca163d5fdd4227a4d0f79e843f13d8; no voters:
I20260501 02:09:03.313637 4744 raft_consensus.cc:2804] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:03.313702 4744 raft_consensus.cc:493] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:03.313733 4744 raft_consensus.cc:3060] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:03.314733 4744 raft_consensus.cc:515] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.315114 4744 leader_election.cc:290] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [CANDIDATE]: Term 1 election: Requested vote from peers 088d19fbc5b84d4eb7962f49561d5fc0 (127.18.96.195:42899), 7696efcd193945de9eeb2f2139fc0e1e (127.18.96.193:35921)
I20260501 02:09:03.315451 4673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a38b3b4d1db40cf89d662dea4d97730" candidate_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "088d19fbc5b84d4eb7962f49561d5fc0"
I20260501 02:09:03.315608 4673 raft_consensus.cc:3060] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:03.315690 4401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a38b3b4d1db40cf89d662dea4d97730" candidate_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7696efcd193945de9eeb2f2139fc0e1e"
I20260501 02:09:03.315860 4401 raft_consensus.cc:3060] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:03.316619 4673 raft_consensus.cc:2468] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ca163d5fdd4227a4d0f79e843f13d8 in term 1.
I20260501 02:09:03.316931 4470 leader_election.cc:304] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 088d19fbc5b84d4eb7962f49561d5fc0, e6ca163d5fdd4227a4d0f79e843f13d8; no voters:
I20260501 02:09:03.316978 4401 raft_consensus.cc:2468] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ca163d5fdd4227a4d0f79e843f13d8 in term 1.
I20260501 02:09:03.317096 4744 raft_consensus.cc:2804] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:03.317332 4744 raft_consensus.cc:697] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 1 LEADER]: Becoming Leader. State: Replica: e6ca163d5fdd4227a4d0f79e843f13d8, State: Running, Role: LEADER
I20260501 02:09:03.317646 4744 consensus_queue.cc:237] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } }
I20260501 02:09:03.320744 4252 catalog_manager.cc:5671] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ca163d5fdd4227a4d0f79e843f13d8 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ca163d5fdd4227a4d0f79e843f13d8" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 36665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 } health_report { overall_health: UNKNOWN } } }
I20260501 02:09:03.338068 4735 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.335331 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.18.96.194:36665)
0501 02:09:03.335462 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.335466 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.335478 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:09:03.335604 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.335607 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.335657 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.335733 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:03.336065 (+ 332us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.336545 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.337145 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.337285 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.337351 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:03.337673 (+ 322us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:03.337752 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.337839 (+ 87us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.337877 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:09:03.343024 4673 raft_consensus.cc:1275] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 1 FOLLOWER]: Refusing update from remote peer e6ca163d5fdd4227a4d0f79e843f13d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:03.343538 4401 raft_consensus.cc:1275] T 3a38b3b4d1db40cf89d662dea4d97730 P 7696efcd193945de9eeb2f2139fc0e1e [term 1 FOLLOWER]: Refusing update from remote peer e6ca163d5fdd4227a4d0f79e843f13d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:03.343722 4744 consensus_queue.cc:1048] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "088d19fbc5b84d4eb7962f49561d5fc0" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:03.344111 4744 consensus_queue.cc:1048] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7696efcd193945de9eeb2f2139fc0e1e" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:03.352679 4752 mvcc.cc:204] Tried to move back new op lower bound from 7281055102326722560 to 7281055102231715840. Current Snapshot: MvccSnapshot[applied={T|T < 7281055102326722560}]
I20260501 02:09:03.359711 4252 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:03.359872 4252 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:03.362195 4252 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 82403159aedd4cdcaa41b4efe1eba4ca: Sending DeleteTablet for 3 replicas of tablet 3a38b3b4d1db40cf89d662dea4d97730
I20260501 02:09:03.363008 4381 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:58350
I20260501 02:09:03.363427 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4318
I20260501 02:09:03.363487 4653 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:55836
I20260501 02:09:03.364257 4517 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a38b3b4d1db40cf89d662dea4d97730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:03 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:56046
I20260501 02:09:03.366230 4760 tablet_replica.cc:333] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: stopping tablet replica
I20260501 02:09:03.366472 4760 raft_consensus.cc:2243] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:03.366767 4760 raft_consensus.cc:2272] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:03.367120 4761 tablet_replica.cc:333] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: stopping tablet replica
I20260501 02:09:03.367334 4761 raft_consensus.cc:2243] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:09:03.367671 4761 raft_consensus.cc:2272] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:03.368789 4761 ts_tablet_manager.cc:1916] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:03.370915 4760 ts_tablet_manager.cc:1916] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:03.371038 4761 ts_tablet_manager.cc:1929] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:03.371142 4761 log.cc:1199] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/3a38b3b4d1db40cf89d662dea4d97730
I20260501 02:09:03.371438 4761 ts_tablet_manager.cc:1950] T 3a38b3b4d1db40cf89d662dea4d97730 P e6ca163d5fdd4227a4d0f79e843f13d8: Deleting consensus metadata
W20260501 02:09:03.372119 4241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:35921 (error 108)
I20260501 02:09:03.372548 4238 catalog_manager.cc:5002] TS e6ca163d5fdd4227a4d0f79e843f13d8 (127.18.96.194:36665): tablet 3a38b3b4d1db40cf89d662dea4d97730 (table test-table [id=257643b4980a46319acf459e41937a38]) successfully deleted
I20260501 02:09:03.372933 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4454
I20260501 02:09:03.374402 4732 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.374111 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35921 (local address 127.0.0.1:58354)
0501 02:09:03.374274 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.374324 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:35921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260501 02:09:03.374545 4241 catalog_manager.cc:4729] TS 7696efcd193945de9eeb2f2139fc0e1e (127.18.96.193:35921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a38b3b4d1db40cf89d662dea4d97730: Network error: Client connection negotiation failed: client connection to 127.18.96.193:35921: connect: Connection refused (error 111)
I20260501 02:09:03.374581 4760 ts_tablet_manager.cc:1929] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:03.374660 4760 log.cc:1199] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/3a38b3b4d1db40cf89d662dea4d97730
I20260501 02:09:03.374986 4760 ts_tablet_manager.cc:1950] T 3a38b3b4d1db40cf89d662dea4d97730 P 088d19fbc5b84d4eb7962f49561d5fc0: Deleting consensus metadata
I20260501 02:09:03.375924 4238 catalog_manager.cc:5002] TS 088d19fbc5b84d4eb7962f49561d5fc0 (127.18.96.195:42899): tablet 3a38b3b4d1db40cf89d662dea4d97730 (table test-table [id=257643b4980a46319acf459e41937a38]) successfully deleted
I20260501 02:09:03.378793 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4590
I20260501 02:09:03.384538 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4222
2026-05-01T02:09:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3385 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-05-01T02:09:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:03Z Disabled control of system clock
I20260501 02:09:03.414516 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37467
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:03.511530 4766 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:03.511710 4766 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:03.511752 4766 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:03.514784 4766 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:03.514830 4766 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:03.514846 4766 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:03.514864 4766 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:03.514880 4766 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:03.518435 4766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:37467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:03.519251 4766 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:03.520162 4766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:03.525095 4774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:03.525086 4772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:03.525089 4771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:03.525591 4766 server_base.cc:1061] running on GCE node
I20260501 02:09:03.526108 4766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:03.526898 4766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:03.528087 4766 hybrid_clock.cc:648] HybridClock initialized: now 1777601343528074 us; error 35 us; skew 500 ppm
I20260501 02:09:03.529649 4766 webserver.cc:492] Webserver started at http://127.18.96.254:41785/ using document root <none> and password file <none>
I20260501 02:09:03.530241 4766 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:03.530315 4766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:03.530550 4766 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:03.532080 4766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "deef189b69e9496fa8eaca4516304891"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "f16ecf3144bb73fb4e0396c3845fbddd"
server_key_iv: "a6c6cfb77cdfe993485805d9eb03b025"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.532562 4766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "deef189b69e9496fa8eaca4516304891"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "f16ecf3144bb73fb4e0396c3845fbddd"
server_key_iv: "a6c6cfb77cdfe993485805d9eb03b025"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.535647 4766 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:03.537654 4780 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.538648 4766 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:03.538777 4766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "deef189b69e9496fa8eaca4516304891"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "f16ecf3144bb73fb4e0396c3845fbddd"
server_key_iv: "a6c6cfb77cdfe993485805d9eb03b025"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.538877 4766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:03.559011 4766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:03.559656 4766 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:03.559844 4766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:03.567253 4766 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:37467
I20260501 02:09:03.567269 4832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:37467 every 8 connection(s)
I20260501 02:09:03.568312 4766 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:03.569077 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4766
I20260501 02:09:03.569212 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:03.569460 18819 external_mini_cluster.cc:1468] Setting key db44e51b6e9159d16429bce9ae7597f7
I20260501 02:09:03.571389 4833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:03.576057 4833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: Bootstrap starting.
I20260501 02:09:03.577255 4836 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.570679 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.18.96.254:37467)
0501 02:09:03.571214 (+ 535us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.571222 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.571253 (+ 31us) server_negotiation.cc:408] Connection header received
0501 02:09:03.571914 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.571932 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.572238 (+ 306us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.572541 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:03.573057 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.574270 (+ 1213us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.574953 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.575230 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.575786 (+ 556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.575840 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:03.575850 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:03.576233 (+ 383us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:03.576279 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:03.576293 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.576401 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.576607 (+ 206us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.576758 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
I20260501 02:09:03.578269 4833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:03.579036 4833 log.cc:826] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:03.580652 4833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: No bootstrap required, opened a new log
I20260501 02:09:03.582957 4833 raft_consensus.cc:359] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } }
I20260501 02:09:03.583117 4833 raft_consensus.cc:385] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:03.583149 4833 raft_consensus.cc:740] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deef189b69e9496fa8eaca4516304891, State: Initialized, Role: FOLLOWER
I20260501 02:09:03.583499 4833 consensus_queue.cc:260] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } }
I20260501 02:09:03.583612 4833 raft_consensus.cc:399] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:03.583648 4833 raft_consensus.cc:493] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:03.583710 4833 raft_consensus.cc:3060] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:03.584465 4833 raft_consensus.cc:515] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } }
I20260501 02:09:03.584854 4833 leader_election.cc:304] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deef189b69e9496fa8eaca4516304891; no voters:
I20260501 02:09:03.585173 4833 leader_election.cc:290] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:03.585273 4838 raft_consensus.cc:2804] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:03.585520 4838 raft_consensus.cc:697] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [term 1 LEADER]: Becoming Leader. State: Replica: deef189b69e9496fa8eaca4516304891, State: Running, Role: LEADER
I20260501 02:09:03.585896 4838 consensus_queue.cc:237] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } }
I20260501 02:09:03.586282 4833 sys_catalog.cc:565] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:03.587479 4839 sys_catalog.cc:455] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deef189b69e9496fa8eaca4516304891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } } }
I20260501 02:09:03.588104 4839 sys_catalog.cc:458] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:03.587988 4840 sys_catalog.cc:455] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader deef189b69e9496fa8eaca4516304891. Latest consensus state: current_term: 1 leader_uuid: "deef189b69e9496fa8eaca4516304891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deef189b69e9496fa8eaca4516304891" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 37467 } } }
I20260501 02:09:03.589100 4840 sys_catalog.cc:458] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891 [sys.catalog]: This master's current role is: LEADER
W20260501 02:09:03.590751 4852 catalog_manager.cc:1568] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:03.590811 4852 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:03.590890 4854 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:03.591603 4854 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:03.596311 4854 catalog_manager.cc:1357] Generated new cluster ID: 9e6ec80c0b434b559941e0bb4709f429
I20260501 02:09:03.596370 4854 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:03.628708 4854 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:03.629550 4854 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:03.639875 4854 catalog_manager.cc:6044] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: Generated new TSK 0
I20260501 02:09:03.640556 4854 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:03.654968 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--builtin_ntp_servers=127.18.96.212:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:03.755246 4857 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:03.755506 4857 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:03.755602 4857 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:03.758872 4857 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:03.758957 4857 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:03.759064 4857 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:03.762802 4857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:03.763854 4857 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:03.764933 4857 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:03.770834 4865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:03.770836 4863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:03.771021 4857 server_base.cc:1061] running on GCE node
W20260501 02:09:03.770862 4862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:03.771504 4857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:03.772017 4857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:03.773159 4857 hybrid_clock.cc:648] HybridClock initialized: now 1777601343773136 us; error 36 us; skew 500 ppm
I20260501 02:09:03.774960 4857 webserver.cc:492] Webserver started at http://127.18.96.193:42121/ using document root <none> and password file <none>
I20260501 02:09:03.775550 4857 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:03.775604 4857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:03.775780 4857 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:03.777230 4857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "7193e12bcdaf4af8b7dcc877c9091826"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "4d8ff179f2ec88907d38d611cdeda2c0"
server_key_iv: "d35572575855acc6b3ce7b15456a5a99"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.777638 4857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "7193e12bcdaf4af8b7dcc877c9091826"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "4d8ff179f2ec88907d38d611cdeda2c0"
server_key_iv: "d35572575855acc6b3ce7b15456a5a99"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.780581 4857 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:03.782480 4871 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.783334 4857 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:03.783425 4857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "7193e12bcdaf4af8b7dcc877c9091826"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "4d8ff179f2ec88907d38d611cdeda2c0"
server_key_iv: "d35572575855acc6b3ce7b15456a5a99"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.783490 4857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:03.796157 4857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:03.796643 4857 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:03.796770 4857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:03.797191 4857 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:03.798053 4857 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:03.798094 4857 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.798130 4857 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:03.798146 4857 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.807036 4857 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:40357
I20260501 02:09:03.807071 4984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:40357 every 8 connection(s)
I20260501 02:09:03.807942 4857 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:09:03.810477 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4857
I20260501 02:09:03.810616 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:03.810907 18819 external_mini_cluster.cc:1468] Setting key 67a5db53d8c6a2ba5712fc3be7c788ea
I20260501 02:09:03.812884 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--builtin_ntp_servers=127.18.96.212:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260501 02:09:03.815517 4836 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.809505 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:45755 (local address 127.18.96.254:37467)
0501 02:09:03.809652 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.809656 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.810520 (+ 864us) server_negotiation.cc:408] Connection header received
0501 02:09:03.811525 (+ 1005us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.811529 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.811575 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.811645 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:03.812550 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.813136 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.814034 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.814199 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.814940 (+ 741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.814961 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:03.814968 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:03.814983 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:03.815009 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:03.815021 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.815100 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.815241 (+ 141us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.815297 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:09:03.815905 4987 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.809744 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37467 (local address 127.18.96.193:45755)
0501 02:09:03.810367 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.810411 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.811339 (+ 928us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.811796 (+ 457us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.811804 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.811856 (+ 52us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:03.812407 (+ 551us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.812423 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.813255 (+ 832us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.813262 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.813858 (+ 596us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.813869 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.814086 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.814744 (+ 658us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:03.814746 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:03.814767 (+ 21us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:03.814808 (+ 41us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:03.815108 (+ 300us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:03.815113 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:03.815117 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.815214 (+ 97us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.815321 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":105,"threads_started":1}
I20260501 02:09:03.816891 4985 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37467
I20260501 02:09:03.817134 4985 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:03.817745 4985 heartbeater.cc:507] Master 127.18.96.254:37467 requested a full tablet report, sending...
I20260501 02:09:03.819198 4797 ts_manager.cc:194] Registered new tserver with Master: 7193e12bcdaf4af8b7dcc877c9091826 (127.18.96.193:40357)
I20260501 02:09:03.820354 4797 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.193:45755
W20260501 02:09:03.911749 4988 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:03.911966 4988 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:03.912062 4988 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:03.915333 4988 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:03.915380 4988 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:03.915463 4988 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:03.919183 4988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:03.920190 4988 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:03.921240 4988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:03.927922 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:03.927932 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:03.927906 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:03.928110 4988 server_base.cc:1061] running on GCE node
I20260501 02:09:03.928501 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:03.929075 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:03.930229 4988 hybrid_clock.cc:648] HybridClock initialized: now 1777601343930205 us; error 37 us; skew 500 ppm
I20260501 02:09:03.931860 4988 webserver.cc:492] Webserver started at http://127.18.96.194:37369/ using document root <none> and password file <none>
I20260501 02:09:03.932386 4988 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:03.932439 4988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:03.932682 4988 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:03.934329 4988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "8eb75a46e0584292b691ebb2cce8e488"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "417f8cd8d0fac9d670e78535f7a8df66"
server_key_iv: "6fc32c9635cc7c3b98b2c5cacdab4e40"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.934808 4988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "8eb75a46e0584292b691ebb2cce8e488"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "417f8cd8d0fac9d670e78535f7a8df66"
server_key_iv: "6fc32c9635cc7c3b98b2c5cacdab4e40"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.937880 4988 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:03.940021 5002 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.940917 4988 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:03.941006 4988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "8eb75a46e0584292b691ebb2cce8e488"
format_stamp: "Formatted at 2026-05-01 02:09:03 on dist-test-slave-2j90"
server_key: "417f8cd8d0fac9d670e78535f7a8df66"
server_key_iv: "6fc32c9635cc7c3b98b2c5cacdab4e40"
server_key_version: "encryptionkey@0"
I20260501 02:09:03.941121 4988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:03.960971 4988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:03.961623 4988 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:03.961812 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:03.962395 4988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:03.963328 4988 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:03.963377 4988 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.963447 4988 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:03.963482 4988 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:03.972685 4988 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:46515
I20260501 02:09:03.972716 5115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:46515 every 8 connection(s)
I20260501 02:09:03.973558 4988 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:03.978845 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 4988
I20260501 02:09:03.978960 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:03.979230 18819 external_mini_cluster.cc:1468] Setting key 6b55a6f2fad0e3fc5acdaf1fdd82f54c
I20260501 02:09:03.980667 4836 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.975191 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:58133 (local address 127.18.96.254:37467)
0501 02:09:03.975328 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:03.975331 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:03.976044 (+ 713us) server_negotiation.cc:408] Connection header received
0501 02:09:03.976888 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:03.976892 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:03.976953 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:03.977057 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:03.977837 (+ 780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.978448 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.979156 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.979332 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.980154 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:03.980169 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:03.980173 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:03.980186 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:03.980211 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:03.980220 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:03.980299 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:03.980474 (+ 175us) server_negotiation.cc:300] Negotiation successful
0501 02:09:03.980523 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:09:03.981302 5118 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:03.975404 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37467 (local address 127.18.96.194:58133)
0501 02:09:03.975914 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:03.975942 (+ 28us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:03.976694 (+ 752us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:03.977203 (+ 509us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:03.977211 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:03.977263 (+ 52us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:03.977690 (+ 427us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.977700 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.978572 (+ 872us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:03.978576 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:03.979033 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:03.979049 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:03.979249 (+ 200us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:03.979916 (+ 667us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:03.979918 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:03.979968 (+ 50us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:03.980022 (+ 54us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:03.980319 (+ 297us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:03.980326 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:03.980331 (+ 5us) client_negotiation.cc:770] Sending connection context
0501 02:09:03.980477 (+ 146us) client_negotiation.cc:241] Negotiation successful
0501 02:09:03.980629 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260501 02:09:03.981596 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--builtin_ntp_servers=127.18.96.212:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260501 02:09:03.982709 5116 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37467
I20260501 02:09:03.982939 5116 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:03.983457 5116 heartbeater.cc:507] Master 127.18.96.254:37467 requested a full tablet report, sending...
I20260501 02:09:03.984539 4797 ts_manager.cc:194] Registered new tserver with Master: 8eb75a46e0584292b691ebb2cce8e488 (127.18.96.194:46515)
I20260501 02:09:03.985270 4797 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.194:58133
W20260501 02:09:04.083961 5119 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:04.084174 5119 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:04.084224 5119 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:04.087730 5119 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:04.087803 5119 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:04.087877 5119 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:04.091790 5119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:37467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:04.092836 5119 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:04.093920 5119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260501 02:09:04.100034 5119 server_base.cc:1061] running on GCE node
W20260501 02:09:04.100013 5124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:04.100127 5127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:04.100255 5125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:04.100610 5119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:04.101132 5119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:04.102317 5119 hybrid_clock.cc:648] HybridClock initialized: now 1777601344102311 us; error 30 us; skew 500 ppm
I20260501 02:09:04.103968 5119 webserver.cc:492] Webserver started at http://127.18.96.195:44685/ using document root <none> and password file <none>
I20260501 02:09:04.104552 5119 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:04.104636 5119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:04.104843 5119 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:04.106520 5119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "951deb2640cb46c2b18a53f896965a5f"
format_stamp: "Formatted at 2026-05-01 02:09:04 on dist-test-slave-2j90"
server_key: "d3b3cc1343a80dc45db874faf5178184"
server_key_iv: "f9cfa9e9196e67e746aa08161d8e2694"
server_key_version: "encryptionkey@0"
I20260501 02:09:04.106999 5119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "951deb2640cb46c2b18a53f896965a5f"
format_stamp: "Formatted at 2026-05-01 02:09:04 on dist-test-slave-2j90"
server_key: "d3b3cc1343a80dc45db874faf5178184"
server_key_iv: "f9cfa9e9196e67e746aa08161d8e2694"
server_key_version: "encryptionkey@0"
I20260501 02:09:04.110147 5119 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:04.112205 5133 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:04.113163 5119 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:04.113305 5119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "951deb2640cb46c2b18a53f896965a5f"
format_stamp: "Formatted at 2026-05-01 02:09:04 on dist-test-slave-2j90"
server_key: "d3b3cc1343a80dc45db874faf5178184"
server_key_iv: "f9cfa9e9196e67e746aa08161d8e2694"
server_key_version: "encryptionkey@0"
I20260501 02:09:04.113431 5119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:04.148267 5119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:04.148931 5119 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:04.149121 5119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:04.149675 5119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:04.150696 5119 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:04.150766 5119 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:04.150838 5119 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:04.150890 5119 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:04.160178 5119 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:45549
I20260501 02:09:04.160207 5246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:45549 every 8 connection(s)
I20260501 02:09:04.161124 5119 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:09:04.167802 4836 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.162853 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:36643 (local address 127.18.96.254:37467)
0501 02:09:04.163001 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.163005 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.163789 (+ 784us) server_negotiation.cc:408] Connection header received
0501 02:09:04.164586 (+ 797us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.164590 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.164639 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.164715 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:04.165461 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.165936 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.166595 (+ 659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.166726 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.167383 (+ 657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:04.167399 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:04.167401 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:04.167413 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:04.167432 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:04.167439 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:04.167500 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.167645 (+ 145us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.167691 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:09:04.168318 5249 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.163098 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:37467 (local address 127.18.96.195:36643)
0501 02:09:04.163651 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.163681 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.164407 (+ 726us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.164856 (+ 449us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.164864 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.164917 (+ 53us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:04.165342 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.165351 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.166107 (+ 756us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.166111 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.166492 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.166499 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.166669 (+ 170us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.167173 (+ 504us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:04.167174 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:04.167196 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:04.167237 (+ 41us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:04.167514 (+ 277us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:04.167519 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:04.167523 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.167630 (+ 107us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.167741 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260501 02:09:04.168804 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5119
I20260501 02:09:04.168933 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:04.169239 18819 external_mini_cluster.cc:1468] Setting key f999e639698227ee77925ed0df3dabae
I20260501 02:09:04.169346 5247 heartbeater.cc:344] Connected to a master server at 127.18.96.254:37467
I20260501 02:09:04.169580 5247 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:04.170184 5247 heartbeater.cc:507] Master 127.18.96.254:37467 requested a full tablet report, sending...
I20260501 02:09:04.171355 4796 ts_manager.cc:194] Registered new tserver with Master: 951deb2640cb46c2b18a53f896965a5f (127.18.96.195:45549)
I20260501 02:09:04.171977 4796 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.195:36643
I20260501 02:09:04.183094 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:04.188048 4836 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.184949 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.18.96.254:37467)
0501 02:09:04.185106 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.185109 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.185179 (+ 70us) server_negotiation.cc:408] Connection header received
0501 02:09:04.185274 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.185276 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.185316 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.185395 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:04.185736 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.186270 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.186885 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.187006 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.187535 (+ 529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:04.187548 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:04.187550 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:04.187561 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:04.187578 (+ 17us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:04.187585 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:04.187691 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.187795 (+ 104us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.187857 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:09:04.190503 4796 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:04.193030 4796 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:04.205052 5258 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.200700 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:46515 (local address 127.0.0.1:58798)
0501 02:09:04.201301 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.201339 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.201480 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.202045 (+ 565us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.202053 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.202206 (+ 153us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:04.202528 (+ 322us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.202536 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.203892 (+ 1356us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.203900 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.204516 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.204523 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.204686 (+ 163us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.204736 (+ 50us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.204825 (+ 89us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.204896 (+ 71us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":88,"threads_started":1}
I20260501 02:09:04.206133 5261 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.201328 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:45549 (local address 127.0.0.1:56324)
0501 02:09:04.202207 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.202220 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.202305 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.202736 (+ 431us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.202739 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.202758 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:04.203013 (+ 255us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.203022 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.204644 (+ 1622us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.204651 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.205719 (+ 1068us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.205730 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.205847 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.205860 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.205915 (+ 55us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.205999 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":72,"threads_started":1}
I20260501 02:09:04.206295 5259 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.200867 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.18.96.194:46515)
0501 02:09:04.201253 (+ 386us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.201259 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.201626 (+ 367us) server_negotiation.cc:408] Connection header received
0501 02:09:04.201713 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.201716 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.201844 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.202033 (+ 189us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:04.202664 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.203742 (+ 1078us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.205258 (+ 1516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.205776 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.205940 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.206037 (+ 97us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.206125 (+ 88us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260501 02:09:04.205862 5260 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.200916 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40357 (local address 127.0.0.1:39324)
0501 02:09:04.201572 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.201586 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.201674 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.203046 (+ 1372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.203050 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.203067 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:04.203289 (+ 222us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.203295 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.204725 (+ 1430us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.204731 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.205492 (+ 761us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.205500 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.205629 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.205642 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.205693 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.205741 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":97,"threads_started":1}
I20260501 02:09:04.206666 5262 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.201410 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.18.96.193:40357)
0501 02:09:04.202529 (+ 1119us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.202534 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.202553 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:09:04.202646 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.202650 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.202800 (+ 150us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.202923 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:04.203420 (+ 497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.204584 (+ 1164us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.205621 (+ 1037us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.206197 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.206372 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.206443 (+ 71us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.206525 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":111,"threads_started":1}
I20260501 02:09:04.207629 5263 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.201727 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.18.96.195:45549)
0501 02:09:04.202081 (+ 354us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.202087 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.202315 (+ 228us) server_negotiation.cc:408] Connection header received
0501 02:09:04.202407 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.202414 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.202589 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.202750 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:04.203319 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.204495 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.206674 (+ 2179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.207212 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.207339 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.207413 (+ 74us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.207499 (+ 86us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260501 02:09:04.209278 4919 tablet_service.cc:1511] Processing CreateTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 (DEFAULT_TABLE table=test-table [id=6a7434d172ed4167b875a7ab6f50fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:04.209278 5050 tablet_service.cc:1511] Processing CreateTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 (DEFAULT_TABLE table=test-table [id=6a7434d172ed4167b875a7ab6f50fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:04.210304 4919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11eb0175c794122b999bd7a8a6c3bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:04.210297 5050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11eb0175c794122b999bd7a8a6c3bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:04.210114 5181 tablet_service.cc:1511] Processing CreateTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 (DEFAULT_TABLE table=test-table [id=6a7434d172ed4167b875a7ab6f50fe08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:04.210896 5181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11eb0175c794122b999bd7a8a6c3bb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:04.215993 5264 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Bootstrap starting.
I20260501 02:09:04.216864 5265 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Bootstrap starting.
I20260501 02:09:04.217037 5266 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Bootstrap starting.
I20260501 02:09:04.218343 5264 tablet_bootstrap.cc:654] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:04.218914 5265 tablet_bootstrap.cc:654] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:04.219013 5266 tablet_bootstrap.cc:654] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:04.219079 5264 log.cc:826] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:04.219712 5265 log.cc:826] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:04.219713 5266 log.cc:826] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:04.220794 5264 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: No bootstrap required, opened a new log
I20260501 02:09:04.220990 5264 ts_tablet_manager.cc:1403] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:09:04.221370 5266 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: No bootstrap required, opened a new log
I20260501 02:09:04.221370 5265 tablet_bootstrap.cc:492] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: No bootstrap required, opened a new log
I20260501 02:09:04.221539 5265 ts_tablet_manager.cc:1403] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:04.221546 5266 ts_tablet_manager.cc:1403] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260501 02:09:04.223661 5264 raft_consensus.cc:359] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.223852 5264 raft_consensus.cc:385] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:04.223924 5264 raft_consensus.cc:740] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 951deb2640cb46c2b18a53f896965a5f, State: Initialized, Role: FOLLOWER
I20260501 02:09:04.224359 5264 consensus_queue.cc:260] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.224462 5265 raft_consensus.cc:359] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.224506 5266 raft_consensus.cc:359] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.224680 5266 raft_consensus.cc:385] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:04.224740 5265 raft_consensus.cc:385] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:04.224769 5266 raft_consensus.cc:740] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7193e12bcdaf4af8b7dcc877c9091826, State: Initialized, Role: FOLLOWER
I20260501 02:09:04.224833 5265 raft_consensus.cc:740] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb75a46e0584292b691ebb2cce8e488, State: Initialized, Role: FOLLOWER
I20260501 02:09:04.224973 5247 heartbeater.cc:499] Master 127.18.96.254:37467 was elected leader, sending a full tablet report...
I20260501 02:09:04.225194 5264 ts_tablet_manager.cc:1434] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:09:04.225231 5266 consensus_queue.cc:260] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.225231 5265 consensus_queue.cc:260] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.225889 5116 heartbeater.cc:499] Master 127.18.96.254:37467 was elected leader, sending a full tablet report...
I20260501 02:09:04.226069 5266 ts_tablet_manager.cc:1434] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260501 02:09:04.226423 4985 heartbeater.cc:499] Master 127.18.96.254:37467 was elected leader, sending a full tablet report...
I20260501 02:09:04.226600 5265 ts_tablet_manager.cc:1434] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260501 02:09:04.292690 5271 raft_consensus.cc:493] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:04.292888 5271 raft_consensus.cc:515] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.294100 5271 leader_election.cc:290] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7193e12bcdaf4af8b7dcc877c9091826 (127.18.96.193:40357), 951deb2640cb46c2b18a53f896965a5f (127.18.96.195:45549)
I20260501 02:09:04.298094 5118 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.294238 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40357 (local address 127.18.96.194:34285)
0501 02:09:04.294392 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.294407 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.294494 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.294874 (+ 380us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.294876 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.294889 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:04.295142 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.295148 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.296689 (+ 1541us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.296694 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.297671 (+ 977us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.297683 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.297811 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.297827 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.297878 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.297929 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260501 02:09:04.298326 5273 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.294331 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:45549 (local address 127.18.96.194:54607)
0501 02:09:04.294660 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.294670 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:04.294754 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:04.294993 (+ 239us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:04.294996 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:04.295008 (+ 12us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:04.295245 (+ 237us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.295252 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.296673 (+ 1421us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.296680 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:04.297683 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:04.297689 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.297811 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.297827 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:09:04.297878 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:09:04.297930 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260501 02:09:04.298528 5262 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.294333 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:34285 (local address 127.18.96.193:40357)
0501 02:09:04.294479 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.294485 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.294503 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:09:04.294650 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.294656 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.294744 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.294842 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:04.295289 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.296553 (+ 1264us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.297809 (+ 1256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.298290 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.298320 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.298371 (+ 51us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.298417 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:09:04.298528 5263 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.294519 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:54607 (local address 127.18.96.195:45549)
0501 02:09:04.294643 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.294647 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.294741 (+ 94us) server_negotiation.cc:408] Connection header received
0501 02:09:04.294844 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.294847 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.294897 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.294973 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:04.295429 (+ 456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.296546 (+ 1117us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.297809 (+ 1263us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.298295 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.298320 (+ 25us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.298373 (+ 53us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.298417 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:09:04.299080 5201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11eb0175c794122b999bd7a8a6c3bb3" candidate_uuid: "8eb75a46e0584292b691ebb2cce8e488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951deb2640cb46c2b18a53f896965a5f" is_pre_election: true
I20260501 02:09:04.299124 4939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11eb0175c794122b999bd7a8a6c3bb3" candidate_uuid: "8eb75a46e0584292b691ebb2cce8e488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7193e12bcdaf4af8b7dcc877c9091826" is_pre_election: true
I20260501 02:09:04.299332 4939 raft_consensus.cc:2468] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb75a46e0584292b691ebb2cce8e488 in term 0.
I20260501 02:09:04.299333 5201 raft_consensus.cc:2468] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb75a46e0584292b691ebb2cce8e488 in term 0.
I20260501 02:09:04.299768 5003 leader_election.cc:304] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7193e12bcdaf4af8b7dcc877c9091826, 8eb75a46e0584292b691ebb2cce8e488; no voters:
I20260501 02:09:04.299990 5271 raft_consensus.cc:2804] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:04.300058 5271 raft_consensus.cc:493] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:04.300110 5271 raft_consensus.cc:3060] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:04.300979 5271 raft_consensus.cc:515] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.301297 5271 leader_election.cc:290] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [CANDIDATE]: Term 1 election: Requested vote from peers 7193e12bcdaf4af8b7dcc877c9091826 (127.18.96.193:40357), 951deb2640cb46c2b18a53f896965a5f (127.18.96.195:45549)
I20260501 02:09:04.301546 4939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11eb0175c794122b999bd7a8a6c3bb3" candidate_uuid: "8eb75a46e0584292b691ebb2cce8e488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7193e12bcdaf4af8b7dcc877c9091826"
I20260501 02:09:04.301685 4939 raft_consensus.cc:3060] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:04.301695 5201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11eb0175c794122b999bd7a8a6c3bb3" candidate_uuid: "8eb75a46e0584292b691ebb2cce8e488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "951deb2640cb46c2b18a53f896965a5f"
I20260501 02:09:04.301823 5201 raft_consensus.cc:3060] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:04.302716 4939 raft_consensus.cc:2468] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb75a46e0584292b691ebb2cce8e488 in term 1.
I20260501 02:09:04.302728 5201 raft_consensus.cc:2468] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb75a46e0584292b691ebb2cce8e488 in term 1.
I20260501 02:09:04.303040 5005 leader_election.cc:304] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8eb75a46e0584292b691ebb2cce8e488, 951deb2640cb46c2b18a53f896965a5f; no voters:
I20260501 02:09:04.303236 5271 raft_consensus.cc:2804] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:04.303422 5271 raft_consensus.cc:697] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb75a46e0584292b691ebb2cce8e488, State: Running, Role: LEADER
I20260501 02:09:04.303750 5271 consensus_queue.cc:237] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } }
I20260501 02:09:04.307224 4796 catalog_manager.cc:5671] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb75a46e0584292b691ebb2cce8e488 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "8eb75a46e0584292b691ebb2cce8e488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb75a46e0584292b691ebb2cce8e488" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 46515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 } health_report { overall_health: UNKNOWN } } }
W20260501 02:09:04.308944 4986 tablet.cc:2404] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:04.326737 5259 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:04.323626 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.18.96.194:46515)
0501 02:09:04.323768 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:04.323772 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:04.323786 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:09:04.323944 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:04.323946 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:04.323999 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:04.324096 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:04.324474 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.324976 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:04.325750 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:04.325904 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:04.326023 (+ 119us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:04.326338 (+ 315us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:04.326423 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:04.326516 (+ 93us) server_negotiation.cc:300] Negotiation successful
0501 02:09:04.326553 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:09:04.331807 4939 raft_consensus.cc:1275] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826 [term 1 FOLLOWER]: Refusing update from remote peer 8eb75a46e0584292b691ebb2cce8e488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:04.331807 5201 raft_consensus.cc:1275] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 1 FOLLOWER]: Refusing update from remote peer 8eb75a46e0584292b691ebb2cce8e488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:04.332477 5271 consensus_queue.cc:1048] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7193e12bcdaf4af8b7dcc877c9091826" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:04.332667 5274 consensus_queue.cc:1048] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "951deb2640cb46c2b18a53f896965a5f" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 45549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:04.339449 5278 mvcc.cc:204] Tried to move back new op lower bound from 7281055106376728576 to 7281055106271608832. Current Snapshot: MvccSnapshot[applied={T|T < 7281055106376728576}]
I20260501 02:09:04.340407 5279 mvcc.cc:204] Tried to move back new op lower bound from 7281055106376728576 to 7281055106271608832. Current Snapshot: MvccSnapshot[applied={T|T < 7281055106376728576}]
I20260501 02:09:04.347009 4796 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:04.347162 4796 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:04.349536 4796 catalog_manager.cc:5958] T 00000000000000000000000000000000 P deef189b69e9496fa8eaca4516304891: Sending DeleteTablet for 3 replicas of tablet c11eb0175c794122b999bd7a8a6c3bb3
I20260501 02:09:04.350244 5050 tablet_service.cc:1558] Processing DeleteTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:04 UTC) from {username='slave'} at 127.0.0.1:58798
I20260501 02:09:04.350306 4919 tablet_service.cc:1558] Processing DeleteTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:04 UTC) from {username='slave'} at 127.0.0.1:39324
I20260501 02:09:04.350775 5288 tablet_replica.cc:333] T c11eb0175c794122b999bd7a8a6c3bb3 P 7193e12bcdaf4af8b7dcc877c9091826: stopping tablet replica
I20260501 02:09:04.350986 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4857
I20260501 02:09:04.351025 5181 tablet_service.cc:1558] Processing DeleteTablet for tablet c11eb0175c794122b999bd7a8a6c3bb3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:04 UTC) from {username='slave'} at 127.0.0.1:56324
I20260501 02:09:04.352486 5287 tablet_replica.cc:333] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: stopping tablet replica
I20260501 02:09:04.352757 5287 raft_consensus.cc:2243] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:09:04.353147 5287 raft_consensus.cc:2272] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:04.354139 5289 tablet_replica.cc:333] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: stopping tablet replica
I20260501 02:09:04.354413 5289 raft_consensus.cc:2243] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:04.354480 5287 ts_tablet_manager.cc:1916] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:04.354794 5289 raft_consensus.cc:2272] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:04.356096 5289 ts_tablet_manager.cc:1916] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:04.357858 5287 ts_tablet_manager.cc:1929] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:04.357977 5287 log.cc:1199] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/c11eb0175c794122b999bd7a8a6c3bb3
I20260501 02:09:04.358353 5287 ts_tablet_manager.cc:1950] T c11eb0175c794122b999bd7a8a6c3bb3 P 8eb75a46e0584292b691ebb2cce8e488: Deleting consensus metadata
I20260501 02:09:04.358736 5289 ts_tablet_manager.cc:1929] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:04.358871 5289 log.cc:1199] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/c11eb0175c794122b999bd7a8a6c3bb3
I20260501 02:09:04.359261 5289 ts_tablet_manager.cc:1950] T c11eb0175c794122b999bd7a8a6c3bb3 P 951deb2640cb46c2b18a53f896965a5f: Deleting consensus metadata
I20260501 02:09:04.359495 4783 catalog_manager.cc:5002] TS 8eb75a46e0584292b691ebb2cce8e488 (127.18.96.194:46515): tablet c11eb0175c794122b999bd7a8a6c3bb3 (table test-table [id=6a7434d172ed4167b875a7ab6f50fe08]) successfully deleted
W20260501 02:09:04.359920 4781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:40357 (error 108)
I20260501 02:09:04.360508 4783 catalog_manager.cc:5002] TS 951deb2640cb46c2b18a53f896965a5f (127.18.96.195:45549): tablet c11eb0175c794122b999bd7a8a6c3bb3 (table test-table [id=6a7434d172ed4167b875a7ab6f50fe08]) successfully deleted
W20260501 02:09:04.360997 5260 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:04.360697 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:40357 (local address 127.0.0.1:39328)
0501 02:09:04.360835 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:04.360909 (+ 74us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:40357: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52}
I20260501 02:09:04.361140 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4988
W20260501 02:09:04.361140 4781 catalog_manager.cc:4729] TS 7193e12bcdaf4af8b7dcc877c9091826 (127.18.96.193:40357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c11eb0175c794122b999bd7a8a6c3bb3: Network error: Client connection negotiation failed: client connection to 127.18.96.193:40357: connect: Connection reset by peer (error 104)
I20260501 02:09:04.367208 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5119
I20260501 02:09:04.373055 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 4766
2026-05-01T02:09:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (986 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:04 dist-test-slave-2j90 krb5kdc[5293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:04 dist-test-slave-2j90 krb5kdc[5293](info): set up 2 sockets
May 01 02:09:04 dist-test-slave-2j90 krb5kdc[5293](info): commencing operation
krb5kdc: starting...
W20260501 02:09:06.350347 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:06 dist-test-slave-2j90 krb5kdc[5293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:06.480798 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44877
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:44877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:06.579032 5309 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:06.579231 5309 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:06.579281 5309 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:06.582868 5309 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:06.582922 5309 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:06.582945 5309 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:06.582986 5309 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:06.583020 5309 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:06.587217 5309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:44877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:06.588232 5309 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:06.589020 5309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:06.594080 5314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:06.594098 5317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:06.594257 5309 server_base.cc:1061] running on GCE node
W20260501 02:09:06.594143 5315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:06.594822 5309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:06.595618 5309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:06.596767 5309 hybrid_clock.cc:648] HybridClock initialized: now 1777601346596750 us; error 30 us; skew 500 ppm
May 01 02:09:06 dist-test-slave-2j90 krb5kdc[5293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601346, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:06.599284 5309 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:06.600234 5309 webserver.cc:492] Webserver started at http://127.18.96.254:39747/ using document root <none> and password file <none>
I20260501 02:09:06.600700 5309 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:06.600745 5309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:06.600883 5309 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:06.602422 5309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "b91edee0d56d4a3c95460b39e6ecf284"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "21a29942b30ef0c213612f2326c1445c"
server_key_iv: "584a97b307f3b7f30bfbc8f24c28b2a5"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.602835 5309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "b91edee0d56d4a3c95460b39e6ecf284"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "21a29942b30ef0c213612f2326c1445c"
server_key_iv: "584a97b307f3b7f30bfbc8f24c28b2a5"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.605738 5309 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:06.607599 5324 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:06.608532 5309 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:06.608683 5309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "b91edee0d56d4a3c95460b39e6ecf284"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "21a29942b30ef0c213612f2326c1445c"
server_key_iv: "584a97b307f3b7f30bfbc8f24c28b2a5"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.608776 5309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:06.627096 5309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:06.629289 5309 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:06.629467 5309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:06.636468 5309 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:44877
I20260501 02:09:06.636474 5376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:44877 every 8 connection(s)
I20260501 02:09:06.637418 5309 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:06.639984 5377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:06.644959 5377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: Bootstrap starting.
I20260501 02:09:06.645605 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5309
I20260501 02:09:06.645751 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:06.646059 18819 external_mini_cluster.cc:1468] Setting key 0b88b3689924dae8394b05090ceb6e76
I20260501 02:09:06.647651 5377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:06.648278 5377 log.cc:826] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:06.650200 5377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: No bootstrap required, opened a new log
May 01 02:09:06 dist-test-slave-2j90 krb5kdc[5293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:06.653399 5377 raft_consensus.cc:359] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } }
I20260501 02:09:06.653600 5377 raft_consensus.cc:385] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:06.653684 5377 raft_consensus.cc:740] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91edee0d56d4a3c95460b39e6ecf284, State: Initialized, Role: FOLLOWER
I20260501 02:09:06.654160 5377 consensus_queue.cc:260] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } }
I20260501 02:09:06.654301 5377 raft_consensus.cc:399] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:06.654384 5377 raft_consensus.cc:493] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:06.654496 5377 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:06.655279 5377 raft_consensus.cc:515] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } }
I20260501 02:09:06.655620 5377 leader_election.cc:304] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b91edee0d56d4a3c95460b39e6ecf284; no voters:
I20260501 02:09:06.655905 5377 leader_election.cc:290] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:06.656162 5382 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:06.656385 5382 raft_consensus.cc:697] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [term 1 LEADER]: Becoming Leader. State: Replica: b91edee0d56d4a3c95460b39e6ecf284, State: Running, Role: LEADER
I20260501 02:09:06.656682 5382 consensus_queue.cc:237] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } }
I20260501 02:09:06.657058 5377 sys_catalog.cc:565] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:06.658113 5383 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b91edee0d56d4a3c95460b39e6ecf284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } } }
I20260501 02:09:06.658221 5383 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:06.658533 5384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b91edee0d56d4a3c95460b39e6ecf284. Latest consensus state: current_term: 1 leader_uuid: "b91edee0d56d4a3c95460b39e6ecf284" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b91edee0d56d4a3c95460b39e6ecf284" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 44877 } } }
I20260501 02:09:06.658572 5391 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:06.658676 5384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:06.658836 5380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:06.647148 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.18.96.254:44877)
0501 02:09:06.647528 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:06.647550 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:06.647579 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:06.648104 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:06.648118 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:06.648406 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:06.648698 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:06.649679 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.650532 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:06.651366 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.651605 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:06.654060 (+ 2455us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:06.654077 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:06.654087 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:06.654112 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:06.655922 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:06.656392 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:06.656398 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:06.656403 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:06.656479 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:06.656969 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:06.656972 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:06.656974 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:06.657324 (+ 350us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:06.657547 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:06.657813 (+ 266us) server_negotiation.cc:300] Negotiation successful
0501 02:09:06.658099 (+ 286us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":146,"threads_started":1}
I20260501 02:09:06.661495 5391 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:06.666496 5391 catalog_manager.cc:1357] Generated new cluster ID: 7da94bb1c56943a18bc8d47fef15a682
I20260501 02:09:06.666571 5391 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:06.677095 5391 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:06.678062 5391 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:06.687973 5391 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: Generated new TSK 0
I20260501 02:09:06.688499 5391 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:06.740959 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44877
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:06.841423 5405 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:06.841607 5405 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:06.841658 5405 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:06.844836 5405 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:06.844887 5405 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:06.845012 5405 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:06.848893 5405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:06.849798 5405 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:06.850546 5405 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:06.856408 5410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:06.856400 5413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:06.856382 5411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:06.856712 5405 server_base.cc:1061] running on GCE node
I20260501 02:09:06.857087 5405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:06.857612 5405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:06.858778 5405 hybrid_clock.cc:648] HybridClock initialized: now 1777601346858767 us; error 28 us; skew 500 ppm
May 01 02:09:06 dist-test-slave-2j90 krb5kdc[5293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601346, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:06.861173 5405 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:09:06.862186 5405 webserver.cc:492] Webserver started at http://127.18.96.193:43925/ using document root <none> and password file <none>
I20260501 02:09:06.862694 5405 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:06.862766 5405 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:06.862996 5405 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:06.864573 5405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "00f937c3dbf5444191982aa0ccc2fc13"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "446afee7976d56252be92d451ae46b8e"
server_key_iv: "04839fc7a77ca6891d98f1eeb79a1bbc"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.865036 5405 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "00f937c3dbf5444191982aa0ccc2fc13"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "446afee7976d56252be92d451ae46b8e"
server_key_iv: "04839fc7a77ca6891d98f1eeb79a1bbc"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.868103 5405 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:06.869880 5420 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:06.870827 5405 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:06.870954 5405 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "00f937c3dbf5444191982aa0ccc2fc13"
format_stamp: "Formatted at 2026-05-01 02:09:06 on dist-test-slave-2j90"
server_key: "446afee7976d56252be92d451ae46b8e"
server_key_iv: "04839fc7a77ca6891d98f1eeb79a1bbc"
server_key_version: "encryptionkey@0"
I20260501 02:09:06.871042 5405 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:06.884425 5405 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:06.886742 5405 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:06.886927 5405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:06.887423 5405 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:06.888242 5405 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:06.888312 5405 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:06.888373 5405 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:06.888425 5405 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:06.897351 5405 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44479
I20260501 02:09:06.897396 5533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44479 every 8 connection(s)
I20260501 02:09:06.898283 5405 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
May 01 02:09:06 dist-test-slave-2j90 krb5kdc[5293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601346, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:06.906308 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5405
I20260501 02:09:06.906454 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:06.906726 18819 external_mini_cluster.cc:1468] Setting key 6e40d4cdbd477c0f01c3076f30ce41a4
I20260501 02:09:06.909497 5380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:06.899885 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:51355 (local address 127.18.96.254:44877)
0501 02:09:06.900046 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:06.900051 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:06.900663 (+ 612us) server_negotiation.cc:408] Connection header received
0501 02:09:06.901388 (+ 725us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:06.901391 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:06.901438 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:06.901511 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:06.902911 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.903407 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:06.903972 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.904096 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:06.906067 (+ 1971us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:06.906084 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:06.906086 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:06.906109 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:06.907615 (+ 1506us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:06.908089 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:06.908093 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:06.908095 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:06.908144 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:06.908443 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:06.908448 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:06.908450 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:06.908616 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:06.908738 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:06.909213 (+ 475us) server_negotiation.cc:300] Negotiation successful
0501 02:09:06.909335 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:09:06.910158 5536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:06.900133 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44877 (local address 127.18.96.193:51355)
0501 02:09:06.900529 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:06.900558 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:06.901224 (+ 666us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:06.901652 (+ 428us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:06.901658 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:06.901998 (+ 340us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:06.902790 (+ 792us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:06.902801 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.903505 (+ 704us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:06.903508 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:06.903881 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:06.903886 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:06.904019 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:06.904500 (+ 481us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:06.904516 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:06.905887 (+ 1371us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:06.907740 (+ 1853us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:06.907746 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:06.907756 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:06.907986 (+ 230us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:06.908243 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:06.908251 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:06.908252 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:06.908334 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:06.908737 (+ 403us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:06.908743 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:06.908978 (+ 235us) client_negotiation.cc:770] Sending connection context
0501 02:09:06.909178 (+ 200us) client_negotiation.cc:241] Negotiation successful
0501 02:09:06.909395 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
I20260501 02:09:06.911262 5534 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44877
I20260501 02:09:06.911495 5534 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:06.911979 5534 heartbeater.cc:507] Master 127.18.96.254:44877 requested a full tablet report, sending...
I20260501 02:09:06.913564 5341 ts_manager.cc:194] Registered new tserver with Master: 00f937c3dbf5444191982aa0ccc2fc13 (127.18.96.193:44479)
I20260501 02:09:06.915215 5341 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:51355
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:06.953179 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44877
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:07.050122 5541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:07.050288 5541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:07.050370 5541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:07.053582 5541 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:07.053637 5541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:07.053767 5541 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:07.058135 5541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:07.059113 5541 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:07.059845 5541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:07.065953 5547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:07.066000 5546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:07.065956 5549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:07.066267 5541 server_base.cc:1061] running on GCE node
I20260501 02:09:07.066612 5541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:07.067116 5541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:07.068261 5541 hybrid_clock.cc:648] HybridClock initialized: now 1777601347068232 us; error 40 us; skew 500 ppm
May 01 02:09:07 dist-test-slave-2j90 krb5kdc[5293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601347, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:07.070647 5541 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:09:07.071606 5541 webserver.cc:492] Webserver started at http://127.18.96.194:46115/ using document root <none> and password file <none>
I20260501 02:09:07.072100 5541 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:07.072173 5541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:07.072378 5541 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:07.073930 5541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "3d9e7a0830354b67ac160c49751c69e7"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "ad69b0c16b52b7c5515e674efbf3f07d"
server_key_iv: "ff0588db7f564a70204171b6ff60bb8e"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.074448 5541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "3d9e7a0830354b67ac160c49751c69e7"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "ad69b0c16b52b7c5515e674efbf3f07d"
server_key_iv: "ff0588db7f564a70204171b6ff60bb8e"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.077411 5541 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:09:07.079401 5556 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.080286 5541 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:07.080417 5541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "3d9e7a0830354b67ac160c49751c69e7"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "ad69b0c16b52b7c5515e674efbf3f07d"
server_key_iv: "ff0588db7f564a70204171b6ff60bb8e"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.080523 5541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:07.089653 5541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:07.091828 5541 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:07.092031 5541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:07.092499 5541 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:07.093278 5541 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:07.093348 5541 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.093411 5541 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:07.093459 5541 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.102630 5541 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:45695
I20260501 02:09:07.102659 5669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:45695 every 8 connection(s)
I20260501 02:09:07.103457 5541 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:07.108201 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5541
I20260501 02:09:07.108358 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:07.108656 18819 external_mini_cluster.cc:1468] Setting key 87439aeb41789def7b744d64d1d9da57
May 01 02:09:07 dist-test-slave-2j90 krb5kdc[5293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601347, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:07.115685 5380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.105031 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:49103 (local address 127.18.96.254:44877)
0501 02:09:07.105198 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.105204 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.105839 (+ 635us) server_negotiation.cc:408] Connection header received
0501 02:09:07.106594 (+ 755us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.106597 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.106646 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.106729 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:07.108137 (+ 1408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.108911 (+ 774us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.109650 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.109834 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.111842 (+ 2008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:07.111862 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:07.111866 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:07.111920 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:07.113555 (+ 1635us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.114228 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.114233 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.114234 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.114286 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.114663 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.114666 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.114667 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.114794 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:07.114867 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.115482 (+ 615us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.115571 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:09:07.116456 5672 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.105263 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44877 (local address 127.18.96.194:49103)
0501 02:09:07.105681 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.105718 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.106426 (+ 708us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.106855 (+ 429us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.106863 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.107172 (+ 309us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:07.107978 (+ 806us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.107989 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.109051 (+ 1062us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.109055 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.109513 (+ 458us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.109520 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.109728 (+ 208us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.110315 (+ 587us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:07.110332 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:07.111700 (+ 1368us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:07.113688 (+ 1988us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:07.113697 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:07.113710 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:07.114107 (+ 397us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:07.114404 (+ 297us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:07.114411 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:07.114416 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:07.114550 (+ 134us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:07.114899 (+ 349us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:07.114908 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:07.115196 (+ 288us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.115438 (+ 242us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.115689 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260501 02:09:07.117460 5670 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44877
I20260501 02:09:07.117707 5670 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:09:07.118259 5670 heartbeater.cc:507] Master 127.18.96.254:44877 requested a full tablet report, sending...
I20260501 02:09:07.119177 5341 ts_manager.cc:194] Registered new tserver with Master: 3d9e7a0830354b67ac160c49751c69e7 (127.18.96.194:45695)
I20260501 02:09:07.119740 5341 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:49103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:07.157126 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:44877
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:07.252880 5677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:07.253060 5677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:07.253141 5677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:07.256397 5677 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:07.256449 5677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:07.256575 5677 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:07.260733 5677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:44877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:07.261730 5677 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:07.262497 5677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:07.268841 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:07.268842 5682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:07.268857 5685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:07.269042 5677 server_base.cc:1061] running on GCE node
I20260501 02:09:07.269428 5677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:07.270006 5677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:07.271195 5677 hybrid_clock.cc:648] HybridClock initialized: now 1777601347271179 us; error 41 us; skew 500 ppm
May 01 02:09:07 dist-test-slave-2j90 krb5kdc[5293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601347, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:07.273729 5677 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:09:07.274796 5677 webserver.cc:492] Webserver started at http://127.18.96.195:42323/ using document root <none> and password file <none>
I20260501 02:09:07.275286 5677 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:07.275367 5677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:07.275560 5677 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:07.277118 5677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "23707293e7664147af228dd6eeb7a484"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "b41af9b9a698430f85e75e7fee2804e6"
server_key_iv: "6e0abb2f7063eeabc1d3602b5942c012"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.277593 5677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "23707293e7664147af228dd6eeb7a484"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "b41af9b9a698430f85e75e7fee2804e6"
server_key_iv: "6e0abb2f7063eeabc1d3602b5942c012"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.280614 5677 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:07.282590 5692 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.283437 5677 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:07.283555 5677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "23707293e7664147af228dd6eeb7a484"
format_stamp: "Formatted at 2026-05-01 02:09:07 on dist-test-slave-2j90"
server_key: "b41af9b9a698430f85e75e7fee2804e6"
server_key_iv: "6e0abb2f7063eeabc1d3602b5942c012"
server_key_version: "encryptionkey@0"
I20260501 02:09:07.283645 5677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:07.296365 5677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:07.298947 5677 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:07.299131 5677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:07.299629 5677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:07.300431 5677 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:07.300508 5677 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.300571 5677 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:07.300616 5677 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:07.310235 5677 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:32811
I20260501 02:09:07.310322 5805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:32811 every 8 connection(s)
I20260501 02:09:07.311223 5677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:09:07.311986 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5677
I20260501 02:09:07.312127 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:07.312386 18819 external_mini_cluster.cc:1468] Setting key 9e30d3938cb26925afcd7455c4022ecc
May 01 02:09:07 dist-test-slave-2j90 krb5kdc[5293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601347, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:07.323232 5380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.313061 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:46801 (local address 127.18.96.254:44877)
0501 02:09:07.313249 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.313252 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.314110 (+ 858us) server_negotiation.cc:408] Connection header received
0501 02:09:07.315097 (+ 987us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.315101 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.315164 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.315244 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:07.316741 (+ 1497us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.317254 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.317942 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.318165 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.320336 (+ 2171us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:07.320363 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:07.320366 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:07.320388 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:07.321581 (+ 1193us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.322112 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.322114 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.322116 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.322154 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.322433 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.322436 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.322437 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.322560 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:07.322633 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.323034 (+ 401us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.323118 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:07.323796 5808 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.313497 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:44877 (local address 127.18.96.195:46801)
0501 02:09:07.313933 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.314000 (+ 67us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.314909 (+ 909us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.315386 (+ 477us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.315393 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.315740 (+ 347us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:07.316584 (+ 844us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.316596 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.317406 (+ 810us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.317409 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.317819 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.317826 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.318054 (+ 228us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.318641 (+ 587us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:07.318660 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:07.320197 (+ 1537us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:07.321687 (+ 1490us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:07.321692 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:07.321699 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:07.321932 (+ 233us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:07.322237 (+ 305us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:07.322240 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:07.322242 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:07.322347 (+ 105us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:07.322647 (+ 300us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:07.322650 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:07.322826 (+ 176us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.322980 (+ 154us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.323144 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260501 02:09:07.324813 5806 heartbeater.cc:344] Connected to a master server at 127.18.96.254:44877
I20260501 02:09:07.325094 5806 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:07.325729 5806 heartbeater.cc:507] Master 127.18.96.254:44877 requested a full tablet report, sending...
I20260501 02:09:07.326884 5341 ts_manager.cc:194] Registered new tserver with Master: 23707293e7664147af228dd6eeb7a484 (127.18.96.195:32811)
I20260501 02:09:07.327426 5341 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:46801
I20260501 02:09:07.338106 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:07.346834 5380 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.339885 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.18.96.254:44877)
0501 02:09:07.340035 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.340038 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.340126 (+ 88us) server_negotiation.cc:408] Connection header received
0501 02:09:07.340220 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.340223 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.340259 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.340324 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:07.341183 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.341666 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.342323 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.342450 (+ 127us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.343750 (+ 1300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:07.343764 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:07.343766 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:07.343787 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:07.345116 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.345582 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.345585 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.345586 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.345632 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:07.345865 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:07.345867 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:07.345868 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:07.346054 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:07.346165 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.346459 (+ 294us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.346657 (+ 198us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:07.349491 5341 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:07.351938 5341 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:07.364941 5818 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.360741 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44479 (local address 127.0.0.1:37038)
0501 02:09:07.361618 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.361628 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.361740 (+ 112us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.362238 (+ 498us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.362242 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.362258 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:07.362493 (+ 235us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.362498 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.363523 (+ 1025us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.363529 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.364440 (+ 911us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.364447 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.364551 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.364596 (+ 45us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.364680 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.364742 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":106,"threads_started":1}
I20260501 02:09:07.364956 5820 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.360293 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32811 (local address 127.0.0.1:49342)
0501 02:09:07.361386 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.361400 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.361486 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.361992 (+ 506us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.361997 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.362020 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:07.362296 (+ 276us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.362303 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.363720 (+ 1417us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.363723 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.364566 (+ 843us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.364578 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.364680 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.364695 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.364742 (+ 47us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.364805 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":107,"threads_started":1}
I20260501 02:09:07.365512 5817 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.360086 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:45695 (local address 127.0.0.1:43164)
0501 02:09:07.361160 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.361197 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.361330 (+ 133us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.362437 (+ 1107us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.362440 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.362466 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:07.362726 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.362732 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.364274 (+ 1542us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.364278 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.365136 (+ 858us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.365144 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.365268 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.365285 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.365351 (+ 66us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.365402 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":79,"threads_started":1}
I20260501 02:09:07.365787 5822 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.361069 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.18.96.193:44479)
0501 02:09:07.361457 (+ 388us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.361463 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.361711 (+ 248us) server_negotiation.cc:408] Connection header received
0501 02:09:07.361895 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.361901 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.362092 (+ 191us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.362234 (+ 142us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:07.362600 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.363318 (+ 718us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.364592 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.365396 (+ 804us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.365498 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.365574 (+ 76us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.365659 (+ 85us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260501 02:09:07.365839 5821 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.360922 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.18.96.195:32811)
0501 02:09:07.361575 (+ 653us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.361580 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.361603 (+ 23us) server_negotiation.cc:408] Connection header received
0501 02:09:07.361667 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.361675 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.361816 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.361968 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:07.362422 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.363593 (+ 1171us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.365027 (+ 1434us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.365523 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.365603 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.365653 (+ 50us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.365714 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":121,"threads_started":1}
I20260501 02:09:07.366657 5819 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.360261 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.18.96.194:45695)
0501 02:09:07.362033 (+ 1772us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.362039 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.362057 (+ 18us) server_negotiation.cc:408] Connection header received
0501 02:09:07.362132 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.362138 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.362285 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.362865 (+ 580us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:07.363000 (+ 135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.364137 (+ 1137us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.365767 (+ 1630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.366256 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.366354 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.366406 (+ 52us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.366465 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648,"thread_start_us":147,"threads_started":1}
I20260501 02:09:07.368402 5740 tablet_service.cc:1511] Processing CreateTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 (DEFAULT_TABLE table=test-table [id=57f36568bd8d4fbfa16841e31e925806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:07.368515 5468 tablet_service.cc:1511] Processing CreateTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 (DEFAULT_TABLE table=test-table [id=57f36568bd8d4fbfa16841e31e925806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:07.368906 5604 tablet_service.cc:1511] Processing CreateTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 (DEFAULT_TABLE table=test-table [id=57f36568bd8d4fbfa16841e31e925806]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:07.369287 5740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759ec6b0b7349ed9e8b02238aa226b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:07.369355 5468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759ec6b0b7349ed9e8b02238aa226b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:07.369632 5604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759ec6b0b7349ed9e8b02238aa226b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:07.374184 5823 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Bootstrap starting.
I20260501 02:09:07.375381 5824 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Bootstrap starting.
I20260501 02:09:07.375725 5825 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Bootstrap starting.
I20260501 02:09:07.377018 5823 tablet_bootstrap.cc:654] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:07.377286 5825 tablet_bootstrap.cc:654] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:07.377291 5824 tablet_bootstrap.cc:654] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:07.377882 5823 log.cc:826] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:07.377926 5825 log.cc:826] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:07.378022 5824 log.cc:826] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:07.379420 5823 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: No bootstrap required, opened a new log
I20260501 02:09:07.379434 5824 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: No bootstrap required, opened a new log
I20260501 02:09:07.379621 5823 ts_tablet_manager.cc:1403] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260501 02:09:07.379623 5824 ts_tablet_manager.cc:1403] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:09:07.380522 5825 tablet_bootstrap.cc:492] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: No bootstrap required, opened a new log
I20260501 02:09:07.380687 5825 ts_tablet_manager.cc:1403] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:07.382707 5823 raft_consensus.cc:359] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.382893 5823 raft_consensus.cc:385] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:07.382946 5823 raft_consensus.cc:740] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23707293e7664147af228dd6eeb7a484, State: Initialized, Role: FOLLOWER
I20260501 02:09:07.383179 5824 raft_consensus.cc:359] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.383364 5824 raft_consensus.cc:385] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:07.383417 5824 raft_consensus.cc:740] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f937c3dbf5444191982aa0ccc2fc13, State: Initialized, Role: FOLLOWER
I20260501 02:09:07.383354 5823 consensus_queue.cc:260] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.383808 5824 consensus_queue.cc:260] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.384265 5823 ts_tablet_manager.cc:1434] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260501 02:09:07.384130 5825 raft_consensus.cc:359] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.384407 5825 raft_consensus.cc:385] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:07.384469 5825 raft_consensus.cc:740] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d9e7a0830354b67ac160c49751c69e7, State: Initialized, Role: FOLLOWER
I20260501 02:09:07.384553 5534 heartbeater.cc:499] Master 127.18.96.254:44877 was elected leader, sending a full tablet report...
I20260501 02:09:07.384558 5806 heartbeater.cc:499] Master 127.18.96.254:44877 was elected leader, sending a full tablet report...
I20260501 02:09:07.384850 5825 consensus_queue.cc:260] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.385448 5670 heartbeater.cc:499] Master 127.18.96.254:44877 was elected leader, sending a full tablet report...
I20260501 02:09:07.385682 5824 ts_tablet_manager.cc:1434] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 02:09:07.386790 5825 ts_tablet_manager.cc:1434] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260501 02:09:07.399319 5535 tablet.cc:2404] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:07.562177 5807 tablet.cc:2404] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:07.604490 5671 tablet.cc:2404] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:07.732569 5831 raft_consensus.cc:493] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:07.732761 5831 raft_consensus.cc:515] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.733942 5831 leader_election.cc:290] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23707293e7664147af228dd6eeb7a484 (127.18.96.195:32811), 00f937c3dbf5444191982aa0ccc2fc13 (127.18.96.193:44479)
I20260501 02:09:07.737030 5832 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.734184 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44479 (local address 127.18.96.194:55441)
0501 02:09:07.734641 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.734656 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.734748 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.735023 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.735025 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.735039 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:07.735287 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.735294 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.735994 (+ 700us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.735997 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.736590 (+ 593us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.736596 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.736722 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.736736 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.736786 (+ 50us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.736835 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":82,"threads_started":1}
I20260501 02:09:07.737401 5833 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.734170 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:32811 (local address 127.18.96.194:51135)
0501 02:09:07.734476 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.734491 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:07.734588 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:07.734857 (+ 269us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:07.734860 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:07.734878 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:07.735174 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.735181 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.735957 (+ 776us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.735960 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:07.736576 (+ 616us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:07.736582 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.736712 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.736730 (+ 18us) client_negotiation.cc:770] Sending connection context
0501 02:09:07.736782 (+ 52us) client_negotiation.cc:241] Negotiation successful
0501 02:09:07.736831 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":72,"threads_started":1}
I20260501 02:09:07.737473 5821 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.734337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:51135 (local address 127.18.96.195:32811)
0501 02:09:07.734499 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.734503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.734573 (+ 70us) server_negotiation.cc:408] Connection header received
0501 02:09:07.734691 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.734696 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.734754 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.734831 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:07.735326 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.735843 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.736700 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.737175 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.737213 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.737265 (+ 52us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.737337 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:07.738050 5822 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.734304 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:55441 (local address 127.18.96.193:44479)
0501 02:09:07.734437 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.734441 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.734747 (+ 306us) server_negotiation.cc:408] Connection header received
0501 02:09:07.734868 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.734871 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.734922 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.735004 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:07.735405 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.735862 (+ 457us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.736739 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.737199 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.737232 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.737827 (+ 595us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.737925 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:09:07.738143 5760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759ec6b0b7349ed9e8b02238aa226b0" candidate_uuid: "3d9e7a0830354b67ac160c49751c69e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23707293e7664147af228dd6eeb7a484" is_pre_election: true
I20260501 02:09:07.738415 5760 raft_consensus.cc:2468] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9e7a0830354b67ac160c49751c69e7 in term 0.
I20260501 02:09:07.738854 5488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759ec6b0b7349ed9e8b02238aa226b0" candidate_uuid: "3d9e7a0830354b67ac160c49751c69e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f937c3dbf5444191982aa0ccc2fc13" is_pre_election: true
I20260501 02:09:07.738885 5560 leader_election.cc:304] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23707293e7664147af228dd6eeb7a484, 3d9e7a0830354b67ac160c49751c69e7; no voters:
I20260501 02:09:07.739085 5488 raft_consensus.cc:2468] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d9e7a0830354b67ac160c49751c69e7 in term 0.
I20260501 02:09:07.739110 5831 raft_consensus.cc:2804] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:07.739244 5831 raft_consensus.cc:493] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:07.739315 5831 raft_consensus.cc:3060] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:07.740604 5831 raft_consensus.cc:515] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.740994 5831 leader_election.cc:290] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [CANDIDATE]: Term 1 election: Requested vote from peers 23707293e7664147af228dd6eeb7a484 (127.18.96.195:32811), 00f937c3dbf5444191982aa0ccc2fc13 (127.18.96.193:44479)
I20260501 02:09:07.741267 5760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759ec6b0b7349ed9e8b02238aa226b0" candidate_uuid: "3d9e7a0830354b67ac160c49751c69e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23707293e7664147af228dd6eeb7a484"
I20260501 02:09:07.741376 5760 raft_consensus.cc:3060] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:07.741376 5488 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759ec6b0b7349ed9e8b02238aa226b0" candidate_uuid: "3d9e7a0830354b67ac160c49751c69e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f937c3dbf5444191982aa0ccc2fc13"
I20260501 02:09:07.741461 5488 raft_consensus.cc:3060] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:07.742357 5760 raft_consensus.cc:2468] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9e7a0830354b67ac160c49751c69e7 in term 1.
I20260501 02:09:07.742444 5488 raft_consensus.cc:2468] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d9e7a0830354b67ac160c49751c69e7 in term 1.
I20260501 02:09:07.742662 5560 leader_election.cc:304] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23707293e7664147af228dd6eeb7a484, 3d9e7a0830354b67ac160c49751c69e7; no voters:
I20260501 02:09:07.742839 5831 raft_consensus.cc:2804] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:07.743041 5831 raft_consensus.cc:697] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3d9e7a0830354b67ac160c49751c69e7, State: Running, Role: LEADER
I20260501 02:09:07.743345 5831 consensus_queue.cc:237] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } }
I20260501 02:09:07.746469 5341 catalog_manager.cc:5671] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d9e7a0830354b67ac160c49751c69e7 (127.18.96.194). New cstate: current_term: 1 leader_uuid: "3d9e7a0830354b67ac160c49751c69e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d9e7a0830354b67ac160c49751c69e7" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 45695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 } health_report { overall_health: UNKNOWN } } }
I20260501 02:09:07.813585 5819 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.810212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.18.96.194:45695)
0501 02:09:07.810349 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:07.810353 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:07.810365 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:09:07.810448 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:07.810451 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:07.810503 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:07.810606 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:07.811009 (+ 403us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.811493 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:07.812157 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:07.812295 (+ 138us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:07.812513 (+ 218us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:07.813061 (+ 548us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:07.813244 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:07.813403 (+ 159us) server_negotiation.cc:300] Negotiation successful
0501 02:09:07.813455 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:09:07.819098 5760 raft_consensus.cc:1275] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 1 FOLLOWER]: Refusing update from remote peer 3d9e7a0830354b67ac160c49751c69e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:07.819099 5488 raft_consensus.cc:1275] T 3759ec6b0b7349ed9e8b02238aa226b0 P 00f937c3dbf5444191982aa0ccc2fc13 [term 1 FOLLOWER]: Refusing update from remote peer 3d9e7a0830354b67ac160c49751c69e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:07.819857 5834 consensus_queue.cc:1048] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f937c3dbf5444191982aa0ccc2fc13" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:07.820088 5831 consensus_queue.cc:1048] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23707293e7664147af228dd6eeb7a484" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:07.828949 5839 mvcc.cc:204] Tried to move back new op lower bound from 7281055120659816448 to 7281055120359333888. Current Snapshot: MvccSnapshot[applied={T|T < 7281055120659816448}]
I20260501 02:09:07.830945 5838 mvcc.cc:204] Tried to move back new op lower bound from 7281055120659816448 to 7281055120359333888. Current Snapshot: MvccSnapshot[applied={T|T < 7281055120659816448 or (T in {7281055120659816448})}]
I20260501 02:09:07.834693 5339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:07.834846 5339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:07.837283 5339 catalog_manager.cc:5958] T 00000000000000000000000000000000 P b91edee0d56d4a3c95460b39e6ecf284: Sending DeleteTablet for 3 replicas of tablet 3759ec6b0b7349ed9e8b02238aa226b0
I20260501 02:09:07.838001 5740 tablet_service.cc:1558] Processing DeleteTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:07 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:49342
I20260501 02:09:07.838016 5604 tablet_service.cc:1558] Processing DeleteTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:07 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:43164
I20260501 02:09:07.838105 5468 tablet_service.cc:1558] Processing DeleteTablet for tablet 3759ec6b0b7349ed9e8b02238aa226b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:07 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:37038
I20260501 02:09:07.838457 5847 tablet_replica.cc:333] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: stopping tablet replica
I20260501 02:09:07.838546 5848 tablet_replica.cc:333] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: stopping tablet replica
I20260501 02:09:07.838717 5847 raft_consensus.cc:2243] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:07.838948 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5405
I20260501 02:09:07.838991 5847 raft_consensus.cc:2272] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:07.841805 5848 raft_consensus.cc:2243] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:09:07.842232 5848 raft_consensus.cc:2272] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:07.843163 5847 ts_tablet_manager.cc:1916] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:07.843592 5848 ts_tablet_manager.cc:1916] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:07.846056 5848 ts_tablet_manager.cc:1929] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:07.846203 5848 log.cc:1199] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/3759ec6b0b7349ed9e8b02238aa226b0
I20260501 02:09:07.846257 5847 ts_tablet_manager.cc:1929] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:07.846346 5847 log.cc:1199] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/3759ec6b0b7349ed9e8b02238aa226b0
I20260501 02:09:07.846638 5847 ts_tablet_manager.cc:1950] T 3759ec6b0b7349ed9e8b02238aa226b0 P 23707293e7664147af228dd6eeb7a484: Deleting consensus metadata
I20260501 02:09:07.846719 5848 ts_tablet_manager.cc:1950] T 3759ec6b0b7349ed9e8b02238aa226b0 P 3d9e7a0830354b67ac160c49751c69e7: Deleting consensus metadata
W20260501 02:09:07.847507 5325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:44479 (error 108)
I20260501 02:09:07.847779 5328 catalog_manager.cc:5002] TS 23707293e7664147af228dd6eeb7a484 (127.18.96.195:32811): tablet 3759ec6b0b7349ed9e8b02238aa226b0 (table test-table [id=57f36568bd8d4fbfa16841e31e925806]) successfully deleted
I20260501 02:09:07.848220 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5541
I20260501 02:09:07.848277 5326 catalog_manager.cc:5002] TS 3d9e7a0830354b67ac160c49751c69e7 (127.18.96.194:45695): tablet 3759ec6b0b7349ed9e8b02238aa226b0 (table test-table [id=57f36568bd8d4fbfa16841e31e925806]) successfully deleted
I20260501 02:09:07.849454 5820 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:07.848369 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44479 (local address 127.0.0.1:37046)
0501 02:09:07.849301 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:07.849366 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":819}
W20260501 02:09:07.849665 5325 catalog_manager.cc:4729] TS 00f937c3dbf5444191982aa0ccc2fc13 (127.18.96.193:44479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3759ec6b0b7349ed9e8b02238aa226b0: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44479: connect: Connection refused (error 111)
I20260501 02:09:07.854795 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5677
I20260501 02:09:07.860615 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5309
2026-05-01T02:09:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3488 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-05-01T02:09:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:07Z Disabled control of system clock
I20260501 02:09:07.889653 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:34191
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:34191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:07.999997 5854 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:08.000221 5854 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:08.000330 5854 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:08.005241 5854 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:08.005327 5854 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:08.005349 5854 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:08.005401 5854 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:08.005446 5854 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:08.011442 5854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:34191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:08.012907 5854 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:08.014528 5854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:08.021029 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.021056 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.021286 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:08.022886 5854 server_base.cc:1061] running on GCE node
I20260501 02:09:08.023330 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:08.024282 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:08.025461 5854 hybrid_clock.cc:648] HybridClock initialized: now 1777601348025431 us; error 45 us; skew 500 ppm
I20260501 02:09:08.027164 5854 webserver.cc:492] Webserver started at http://127.18.96.254:45745/ using document root <none> and password file <none>
I20260501 02:09:08.027722 5854 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:08.027799 5854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:08.028009 5854 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:08.029677 5854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "5682eab723e84e59bdeed055c9adb8bd"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "92c8259fb464a8c8990776f6e83caf34"
server_key_iv: "d76fd4cc0deb83b3a9b736a9afdd4ac2"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.030301 5854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "5682eab723e84e59bdeed055c9adb8bd"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "92c8259fb464a8c8990776f6e83caf34"
server_key_iv: "d76fd4cc0deb83b3a9b736a9afdd4ac2"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.033461 5854 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260501 02:09:08.035772 5868 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.036677 5854 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:08.036801 5854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "5682eab723e84e59bdeed055c9adb8bd"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "92c8259fb464a8c8990776f6e83caf34"
server_key_iv: "d76fd4cc0deb83b3a9b736a9afdd4ac2"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.036892 5854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:08.053416 5854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:08.054020 5854 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:08.054211 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:08.061255 5854 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:34191
I20260501 02:09:08.061269 5920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:34191 every 8 connection(s)
I20260501 02:09:08.062201 5854 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:08.064314 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5854
I20260501 02:09:08.064466 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:08.064718 18819 external_mini_cluster.cc:1468] Setting key b8e20fb59e4e82e2b32d5cdcc216851e
I20260501 02:09:08.064899 5921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:08.070497 5921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: Bootstrap starting.
I20260501 02:09:08.072408 5921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:08.073165 5921 log.cc:826] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:08.074891 5924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.065895 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.18.96.254:34191)
0501 02:09:08.066444 (+ 549us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.066452 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.066482 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:09:08.067220 (+ 738us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.067238 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.067538 (+ 300us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.067844 (+ 306us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:08.068337 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.069059 (+ 722us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.069729 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.070043 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.073331 (+ 3288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:08.073354 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:08.073364 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:08.073683 (+ 319us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:08.073715 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:08.073752 (+ 37us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:08.073878 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.074099 (+ 221us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.074226 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260501 02:09:08.075124 5921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: No bootstrap required, opened a new log
I20260501 02:09:08.077641 5921 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } }
I20260501 02:09:08.077795 5921 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:08.077828 5921 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5682eab723e84e59bdeed055c9adb8bd, State: Initialized, Role: FOLLOWER
I20260501 02:09:08.078189 5921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } }
I20260501 02:09:08.078292 5921 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:08.078333 5921 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:08.078393 5921 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:08.079100 5921 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } }
I20260501 02:09:08.079363 5921 leader_election.cc:304] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5682eab723e84e59bdeed055c9adb8bd; no voters:
I20260501 02:09:08.079576 5921 leader_election.cc:290] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:08.079696 5926 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:08.079941 5926 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [term 1 LEADER]: Becoming Leader. State: Replica: 5682eab723e84e59bdeed055c9adb8bd, State: Running, Role: LEADER
I20260501 02:09:08.080199 5926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } }
I20260501 02:09:08.080591 5921 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:08.081707 5928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5682eab723e84e59bdeed055c9adb8bd. Latest consensus state: current_term: 1 leader_uuid: "5682eab723e84e59bdeed055c9adb8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } } }
I20260501 02:09:08.081817 5928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:08.082626 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5682eab723e84e59bdeed055c9adb8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5682eab723e84e59bdeed055c9adb8bd" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34191 } } }
I20260501 02:09:08.082719 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:08.082829 5935 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:08.085582 5935 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:08.090128 5935 catalog_manager.cc:1357] Generated new cluster ID: bba32c9a74d44f3298652d256841b793
I20260501 02:09:08.090191 5935 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:08.099211 5935 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:08.099922 5935 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:08.114392 5935 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: Generated new TSK 0
I20260501 02:09:08.114902 5935 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:08.123118 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--builtin_ntp_servers=127.18.96.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:08.225175 5945 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:08.225412 5945 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:08.225517 5945 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:08.228808 5945 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:08.228878 5945 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:08.228971 5945 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:08.232698 5945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:08.233705 5945 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:08.234803 5945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:08.240710 5951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.240720 5950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.240921 5953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:08.241106 5945 server_base.cc:1061] running on GCE node
I20260501 02:09:08.241483 5945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:08.242060 5945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:08.243228 5945 hybrid_clock.cc:648] HybridClock initialized: now 1777601348243218 us; error 26 us; skew 500 ppm
I20260501 02:09:08.244858 5945 webserver.cc:492] Webserver started at http://127.18.96.193:43623/ using document root <none> and password file <none>
I20260501 02:09:08.245400 5945 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:08.245481 5945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:08.245689 5945 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:08.247283 5945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "240f12b65a974b1ba74c99da1e63f3e7"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "ae471d575490285201d7b2b4e074f7aa"
server_key_iv: "96980351b35e342ddd822626a687c844"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.247749 5945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "240f12b65a974b1ba74c99da1e63f3e7"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "ae471d575490285201d7b2b4e074f7aa"
server_key_iv: "96980351b35e342ddd822626a687c844"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.250762 5945 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260501 02:09:08.252696 5959 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.253540 5945 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:09:08.253669 5945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "240f12b65a974b1ba74c99da1e63f3e7"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "ae471d575490285201d7b2b4e074f7aa"
server_key_iv: "96980351b35e342ddd822626a687c844"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.253765 5945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:08.274339 5945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:08.274986 5945 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:08.275183 5945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:08.275733 5945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:08.276657 5945 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:08.276728 5945 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.276798 5945 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:08.276839 5945 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.286198 5945 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:44013
I20260501 02:09:08.286267 6072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:44013 every 8 connection(s)
I20260501 02:09:08.287117 5945 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:09:08.289054 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 5945
I20260501 02:09:08.289176 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:08.289476 18819 external_mini_cluster.cc:1468] Setting key 846d377d7eba02782bfd989eca5edd80
I20260501 02:09:08.291729 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--builtin_ntp_servers=127.18.96.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260501 02:09:08.295410 5924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.288844 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:45941 (local address 127.18.96.254:34191)
0501 02:09:08.288985 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.288989 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.290001 (+ 1012us) server_negotiation.cc:408] Connection header received
0501 02:09:08.291112 (+ 1111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.291116 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.291177 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.291276 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:08.292193 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.292978 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.293915 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.294164 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.294811 (+ 647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:08.294832 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:08.294839 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:08.294855 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:08.294881 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:08.294892 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:08.294978 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.295162 (+ 184us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.295229 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:09:08.295946 6075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.289205 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:34191 (local address 127.18.96.193:45941)
0501 02:09:08.289816 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.289872 (+ 56us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.290835 (+ 963us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.291440 (+ 605us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.291452 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.291527 (+ 75us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:08.292053 (+ 526us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.292064 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.293121 (+ 1057us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.293127 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.293778 (+ 651us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.293790 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.294027 (+ 237us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.294606 (+ 579us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:08.294608 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:08.294632 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:08.294673 (+ 41us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:08.294983 (+ 310us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:08.294988 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:08.294992 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.295153 (+ 161us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.295266 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260501 02:09:08.297006 6073 heartbeater.cc:344] Connected to a master server at 127.18.96.254:34191
I20260501 02:09:08.297235 6073 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:08.297734 6073 heartbeater.cc:507] Master 127.18.96.254:34191 requested a full tablet report, sending...
I20260501 02:09:08.299350 5882 ts_manager.cc:194] Registered new tserver with Master: 240f12b65a974b1ba74c99da1e63f3e7 (127.18.96.193:44013)
I20260501 02:09:08.301075 5882 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.193:45941
W20260501 02:09:08.391414 6076 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:08.391592 6076 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:08.391636 6076 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:08.394752 6076 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:08.394795 6076 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:08.394860 6076 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:08.398536 6076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:08.399417 6076 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:08.400380 6076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:08.406309 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.406281 6082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.406363 6081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:08.406597 6076 server_base.cc:1061] running on GCE node
I20260501 02:09:08.406970 6076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:08.407550 6076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:08.408712 6076 hybrid_clock.cc:648] HybridClock initialized: now 1777601348408698 us; error 31 us; skew 500 ppm
I20260501 02:09:08.410431 6076 webserver.cc:492] Webserver started at http://127.18.96.194:43801/ using document root <none> and password file <none>
I20260501 02:09:08.410974 6076 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:08.411031 6076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:08.411276 6076 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:08.412896 6076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "eee432c4046a465a9d4d8f3666f95979"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "1ff77ad253b8384f68fee2c04daf3f19"
server_key_iv: "7600da4a515386ce28591556c7833499"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.413381 6076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "eee432c4046a465a9d4d8f3666f95979"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "1ff77ad253b8384f68fee2c04daf3f19"
server_key_iv: "7600da4a515386ce28591556c7833499"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.416743 6076 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:08.418869 6090 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.419690 6076 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:08.419816 6076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "eee432c4046a465a9d4d8f3666f95979"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "1ff77ad253b8384f68fee2c04daf3f19"
server_key_iv: "7600da4a515386ce28591556c7833499"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.419907 6076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:08.435550 6076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:08.436089 6076 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:08.436268 6076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:08.436745 6076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:08.437664 6076 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:08.437731 6076 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.437801 6076 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:08.437844 6076 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.446945 6076 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:39763
I20260501 02:09:08.446978 6203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:39763 every 8 connection(s)
I20260501 02:09:08.447796 6076 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:08.453977 5924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.449330 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:41891 (local address 127.18.96.254:34191)
0501 02:09:08.449482 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.449485 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.450223 (+ 738us) server_negotiation.cc:408] Connection header received
0501 02:09:08.450976 (+ 753us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.450979 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.451028 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.451099 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:08.451812 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.452277 (+ 465us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.452828 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.452965 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.453567 (+ 602us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:08.453584 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:08.453587 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:08.453597 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:08.453615 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:08.453623 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:08.453682 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.453810 (+ 128us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.453852 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260501 02:09:08.454532 6206 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.449548 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:34191 (local address 127.18.96.194:41891)
0501 02:09:08.450099 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.450128 (+ 29us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.450817 (+ 689us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.451224 (+ 407us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.451231 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.451281 (+ 50us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:08.451699 (+ 418us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.451708 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.452382 (+ 674us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.452385 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.452737 (+ 352us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.452742 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.452876 (+ 134us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.453386 (+ 510us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:08.453388 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:08.453410 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:08.453451 (+ 41us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:08.453695 (+ 244us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:08.453700 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:08.453704 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.453797 (+ 93us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.453896 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260501 02:09:08.455511 6204 heartbeater.cc:344] Connected to a master server at 127.18.96.254:34191
I20260501 02:09:08.455700 6204 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:08.456110 6204 heartbeater.cc:507] Master 127.18.96.254:34191 requested a full tablet report, sending...
I20260501 02:09:08.456954 5882 ts_manager.cc:194] Registered new tserver with Master: eee432c4046a465a9d4d8f3666f95979 (127.18.96.194:39763)
I20260501 02:09:08.457594 5882 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.194:41891
I20260501 02:09:08.457672 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6076
I20260501 02:09:08.457777 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:08.458115 18819 external_mini_cluster.cc:1468] Setting key 35dd50f87992126542d4c8ea67851533
I20260501 02:09:08.459746 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--builtin_ntp_servers=127.18.96.212:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:08.552953 6207 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:08.553128 6207 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:08.553205 6207 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:08.556609 6207 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:08.556658 6207 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:08.556763 6207 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:08.560532 6207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:08.561432 6207 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:08.562502 6207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:08.568243 6212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.568238 6215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:08.568487 6213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:08.568688 6207 server_base.cc:1061] running on GCE node
I20260501 02:09:08.569109 6207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:08.569730 6207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:08.570997 6207 hybrid_clock.cc:648] HybridClock initialized: now 1777601348570941 us; error 69 us; skew 500 ppm
I20260501 02:09:08.572718 6207 webserver.cc:492] Webserver started at http://127.18.96.195:43229/ using document root <none> and password file <none>
I20260501 02:09:08.573274 6207 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:08.573354 6207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:08.573563 6207 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:08.575229 6207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "c9017cda4453e740043d4876bba60592"
server_key_iv: "1f80c357c693f5083d0375509d31dc2f"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.575708 6207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "c9017cda4453e740043d4876bba60592"
server_key_iv: "1f80c357c693f5083d0375509d31dc2f"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.578774 6207 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:09:08.580830 6221 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.581808 6207 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:09:08.581943 6207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2"
format_stamp: "Formatted at 2026-05-01 02:09:08 on dist-test-slave-2j90"
server_key: "c9017cda4453e740043d4876bba60592"
server_key_iv: "1f80c357c693f5083d0375509d31dc2f"
server_key_version: "encryptionkey@0"
I20260501 02:09:08.582129 6207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:08.604138 6207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:08.604820 6207 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:08.605018 6207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:08.605576 6207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:08.606633 6207 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:08.606712 6207 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.606788 6207 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:08.606833 6207 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:08.616663 6207 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:43151
I20260501 02:09:08.617056 6334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:43151 every 8 connection(s)
I20260501 02:09:08.617964 6207 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:09:08.624891 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6207
I20260501 02:09:08.625031 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:08.625344 18819 external_mini_cluster.cc:1468] Setting key e32b56f06e79cd6a2e17625c918c2fb8
I20260501 02:09:08.626842 5924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.620085 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:58579 (local address 127.18.96.254:34191)
0501 02:09:08.620293 (+ 208us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.620299 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.621240 (+ 941us) server_negotiation.cc:408] Connection header received
0501 02:09:08.622521 (+ 1281us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.622526 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.622633 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.622756 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:08.623810 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.624477 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.625238 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.625467 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.626346 (+ 879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:08.626367 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:08.626369 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:08.626381 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:08.626399 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:08.626406 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:08.626502 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.626677 (+ 175us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.626734 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260501 02:09:08.627549 6337 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.620397 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:34191 (local address 127.18.96.195:58579)
0501 02:09:08.621104 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.621142 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.622237 (+ 1095us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.622924 (+ 687us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.622938 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.623004 (+ 66us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:08.623669 (+ 665us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.623686 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.624611 (+ 925us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.624617 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.625109 (+ 492us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.625118 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.625320 (+ 202us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.626111 (+ 791us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:08.626113 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:08.626136 (+ 23us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:08.626179 (+ 43us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:08.626521 (+ 342us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:08.626528 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:08.626532 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.626646 (+ 114us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.626803 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":155,"threads_started":1}
I20260501 02:09:08.628456 6335 heartbeater.cc:344] Connected to a master server at 127.18.96.254:34191
I20260501 02:09:08.628674 6335 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:08.629110 6335 heartbeater.cc:507] Master 127.18.96.254:34191 requested a full tablet report, sending...
I20260501 02:09:08.630002 5882 ts_manager.cc:194] Registered new tserver with Master: 7fadf05c13a34040a8f5d8c58fb1cbd2 (127.18.96.195:43151)
I20260501 02:09:08.630509 5882 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.195:58579
I20260501 02:09:08.639398 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:08.644526 5924 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.641290 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.18.96.254:34191)
0501 02:09:08.641406 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.641409 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.641628 (+ 219us) server_negotiation.cc:408] Connection header received
0501 02:09:08.641722 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.641725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.641762 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.641827 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:08.642303 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.642763 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.643444 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.643558 (+ 114us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.644104 (+ 546us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:08.644117 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:08.644119 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:08.644128 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:08.644144 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:08.644151 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:08.644249 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.644405 (+ 156us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.644430 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:09:08.646932 5882 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:08.649264 5882 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:08.660998 6346 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.657143 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:39763 (local address 127.0.0.1:35606)
0501 02:09:08.657413 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.657448 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.657622 (+ 174us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.658356 (+ 734us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.658362 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.658386 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:08.658651 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.658660 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.659950 (+ 1290us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.659953 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.660584 (+ 631us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.660591 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.660698 (+ 107us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.660736 (+ 38us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.660818 (+ 82us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.660880 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
I20260501 02:09:08.661738 6348 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.657359 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44013 (local address 127.0.0.1:32868)
0501 02:09:08.657683 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.657696 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.657766 (+ 70us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.658818 (+ 1052us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.658821 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.658835 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:08.659079 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.659084 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.660580 (+ 1496us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.660583 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.661376 (+ 793us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.661396 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.661505 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.661517 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.661561 (+ 44us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.661626 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260501 02:09:08.661795 6347 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.657275 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.18.96.194:39763)
0501 02:09:08.657833 (+ 558us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.657842 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.657868 (+ 26us) server_negotiation.cc:408] Connection header received
0501 02:09:08.657984 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.657991 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.658180 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.658816 (+ 636us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:08.658952 (+ 136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.659810 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.660709 (+ 899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.661337 (+ 628us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.661488 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.661547 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.661658 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":111,"threads_started":1}
I20260501 02:09:08.662631 6349 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.657765 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.18.96.193:44013)
0501 02:09:08.658356 (+ 591us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.658361 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.658381 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:09:08.658459 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.658469 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.658649 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.659228 (+ 579us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:08.659360 (+ 132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.660463 (+ 1103us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.661495 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.662249 (+ 754us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.662365 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.662420 (+ 55us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.662486 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":106,"threads_started":1}
I20260501 02:09:08.663110 6350 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.658388 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43151 (local address 127.0.0.1:34256)
0501 02:09:08.658613 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.658625 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.658724 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.659740 (+ 1016us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.659744 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.659767 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:08.659988 (+ 221us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.659994 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.662177 (+ 2183us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.662182 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.662788 (+ 606us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.662796 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.662898 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.662910 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.662955 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.663001 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260501 02:09:08.664211 6351 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.658982 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.18.96.195:43151)
0501 02:09:08.659354 (+ 372us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.659359 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.659379 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:09:08.659462 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.659467 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.659603 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.659724 (+ 121us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:08.660241 (+ 517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.662078 (+ 1837us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.663373 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.663819 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.663929 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.663986 (+ 57us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.664052 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260501 02:09:08.664891 6007 tablet_service.cc:1511] Processing CreateTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc (DEFAULT_TABLE table=test-table [id=690cf996564d4cedae111bc412ce759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:08.664892 6138 tablet_service.cc:1511] Processing CreateTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc (DEFAULT_TABLE table=test-table [id=690cf996564d4cedae111bc412ce759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:08.665745 6007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0893b63071d444e9ac3872e9a11aa8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:08.665776 6138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0893b63071d444e9ac3872e9a11aa8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:08.666783 6269 tablet_service.cc:1511] Processing CreateTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc (DEFAULT_TABLE table=test-table [id=690cf996564d4cedae111bc412ce759f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:08.667520 6269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0893b63071d444e9ac3872e9a11aa8bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:08.670688 6352 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: Bootstrap starting.
I20260501 02:09:08.671713 6353 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Bootstrap starting.
I20260501 02:09:08.672653 6352 tablet_bootstrap.cc:654] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:08.672878 6354 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Bootstrap starting.
I20260501 02:09:08.673681 6353 tablet_bootstrap.cc:654] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:08.673784 6352 log.cc:826] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:08.674505 6353 log.cc:826] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:08.674880 6354 tablet_bootstrap.cc:654] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:08.675578 6354 log.cc:826] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:08.675632 6352 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: No bootstrap required, opened a new log
I20260501 02:09:08.675782 6352 ts_tablet_manager.cc:1403] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:08.675977 6353 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: No bootstrap required, opened a new log
I20260501 02:09:08.676146 6353 ts_tablet_manager.cc:1403] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:08.677125 6354 tablet_bootstrap.cc:492] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: No bootstrap required, opened a new log
I20260501 02:09:08.677289 6354 ts_tablet_manager.cc:1403] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:08.678349 6353 raft_consensus.cc:359] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.678534 6353 raft_consensus.cc:385] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:08.678586 6353 raft_consensus.cc:740] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee432c4046a465a9d4d8f3666f95979, State: Initialized, Role: FOLLOWER
I20260501 02:09:08.678704 6352 raft_consensus.cc:359] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.678905 6352 raft_consensus.cc:385] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:08.678951 6352 raft_consensus.cc:740] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240f12b65a974b1ba74c99da1e63f3e7, State: Initialized, Role: FOLLOWER
I20260501 02:09:08.679023 6353 consensus_queue.cc:260] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.679329 6352 consensus_queue.cc:260] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.679826 6353 ts_tablet_manager.cc:1434] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260501 02:09:08.679949 6073 heartbeater.cc:499] Master 127.18.96.254:34191 was elected leader, sending a full tablet report...
I20260501 02:09:08.680075 6204 heartbeater.cc:499] Master 127.18.96.254:34191 was elected leader, sending a full tablet report...
I20260501 02:09:08.680102 6352 ts_tablet_manager.cc:1434] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260501 02:09:08.680017 6354 raft_consensus.cc:359] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.680186 6354 raft_consensus.cc:385] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:08.680243 6354 raft_consensus.cc:740] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fadf05c13a34040a8f5d8c58fb1cbd2, State: Initialized, Role: FOLLOWER
I20260501 02:09:08.680584 6354 consensus_queue.cc:260] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.681133 6335 heartbeater.cc:499] Master 127.18.96.254:34191 was elected leader, sending a full tablet report...
I20260501 02:09:08.681308 6354 ts_tablet_manager.cc:1434] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260501 02:09:08.698861 6205 tablet.cc:2404] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:08.727712 6359 raft_consensus.cc:493] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:08.727917 6359 raft_consensus.cc:515] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.729007 6359 leader_election.cc:290] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee432c4046a465a9d4d8f3666f95979 (127.18.96.194:39763), 7fadf05c13a34040a8f5d8c58fb1cbd2 (127.18.96.195:43151)
I20260501 02:09:08.731850 6075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.729243 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:39763 (local address 127.18.96.193:40099)
0501 02:09:08.729465 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.729480 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.729574 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.729845 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.729848 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.729868 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:08.730147 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.730155 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.730890 (+ 735us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.730895 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.731472 (+ 577us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.731479 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.731607 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.731624 (+ 17us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.731675 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.731725 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260501 02:09:08.732246 6361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.729243 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:43151 (local address 127.18.96.193:49331)
0501 02:09:08.729661 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.729675 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:08.729784 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:08.730083 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:08.730086 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:08.730103 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:08.730323 (+ 220us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.730328 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.731272 (+ 944us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.731280 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:08.731846 (+ 566us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:08.731852 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.731980 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.731996 (+ 16us) client_negotiation.cc:770] Sending connection context
0501 02:09:08.732050 (+ 54us) client_negotiation.cc:241] Negotiation successful
0501 02:09:08.732100 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":35,"thread_start_us":140,"threads_started":1}
I20260501 02:09:08.732272 6347 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.729364 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:40099 (local address 127.18.96.194:39763)
0501 02:09:08.729480 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.729484 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.729571 (+ 87us) server_negotiation.cc:408] Connection header received
0501 02:09:08.729678 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.729680 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.729730 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.729829 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:08.730267 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.730774 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.731594 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.732052 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.732081 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.732127 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.732177 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260501 02:09:08.732836 6351 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.729378 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:49331 (local address 127.18.96.195:43151)
0501 02:09:08.729511 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.729516 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.729759 (+ 243us) server_negotiation.cc:408] Connection header received
0501 02:09:08.729901 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.729905 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.729975 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.730069 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:08.730440 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.731134 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.732085 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.732568 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.732599 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.732646 (+ 47us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.732696 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:09:08.732983 6158 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0893b63071d444e9ac3872e9a11aa8bc" candidate_uuid: "240f12b65a974b1ba74c99da1e63f3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee432c4046a465a9d4d8f3666f95979" is_pre_election: true
I20260501 02:09:08.733263 6158 raft_consensus.cc:2468] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240f12b65a974b1ba74c99da1e63f3e7 in term 0.
I20260501 02:09:08.733366 6289 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0893b63071d444e9ac3872e9a11aa8bc" candidate_uuid: "240f12b65a974b1ba74c99da1e63f3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" is_pre_election: true
I20260501 02:09:08.733582 6289 raft_consensus.cc:2468] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240f12b65a974b1ba74c99da1e63f3e7 in term 0.
I20260501 02:09:08.733731 5961 leader_election.cc:304] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240f12b65a974b1ba74c99da1e63f3e7, eee432c4046a465a9d4d8f3666f95979; no voters:
I20260501 02:09:08.734021 6359 raft_consensus.cc:2804] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:08.734097 6359 raft_consensus.cc:493] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:08.734128 6359 raft_consensus.cc:3060] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:08.735049 6359 raft_consensus.cc:515] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.735441 6359 leader_election.cc:290] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [CANDIDATE]: Term 1 election: Requested vote from peers eee432c4046a465a9d4d8f3666f95979 (127.18.96.194:39763), 7fadf05c13a34040a8f5d8c58fb1cbd2 (127.18.96.195:43151)
I20260501 02:09:08.735813 6158 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0893b63071d444e9ac3872e9a11aa8bc" candidate_uuid: "240f12b65a974b1ba74c99da1e63f3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee432c4046a465a9d4d8f3666f95979"
I20260501 02:09:08.735813 6289 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0893b63071d444e9ac3872e9a11aa8bc" candidate_uuid: "240f12b65a974b1ba74c99da1e63f3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2"
I20260501 02:09:08.735939 6289 raft_consensus.cc:3060] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:08.735931 6158 raft_consensus.cc:3060] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:08.737107 6289 raft_consensus.cc:2468] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240f12b65a974b1ba74c99da1e63f3e7 in term 1.
I20260501 02:09:08.737231 6158 raft_consensus.cc:2468] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240f12b65a974b1ba74c99da1e63f3e7 in term 1.
I20260501 02:09:08.737421 5963 leader_election.cc:304] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240f12b65a974b1ba74c99da1e63f3e7, 7fadf05c13a34040a8f5d8c58fb1cbd2; no voters:
I20260501 02:09:08.737589 6359 raft_consensus.cc:2804] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:08.737880 6359 raft_consensus.cc:697] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 240f12b65a974b1ba74c99da1e63f3e7, State: Running, Role: LEADER
I20260501 02:09:08.738178 6359 consensus_queue.cc:237] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } }
I20260501 02:09:08.741575 5882 catalog_manager.cc:5671] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 240f12b65a974b1ba74c99da1e63f3e7 (127.18.96.193). New cstate: current_term: 1 leader_uuid: "240f12b65a974b1ba74c99da1e63f3e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "240f12b65a974b1ba74c99da1e63f3e7" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 } health_report { overall_health: UNKNOWN } } }
I20260501 02:09:08.759626 6349 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.756626 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.18.96.193:44013)
0501 02:09:08.756747 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:08.756750 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:08.756763 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:09:08.756888 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:08.756891 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:08.756940 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:08.757018 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:08.757370 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.757871 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:08.758557 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:08.758678 (+ 121us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:08.758879 (+ 201us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:08.759185 (+ 306us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:08.759289 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:08.759431 (+ 142us) server_negotiation.cc:300] Negotiation successful
0501 02:09:08.759474 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:09:08.764714 6158 raft_consensus.cc:1275] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 1 FOLLOWER]: Refusing update from remote peer 240f12b65a974b1ba74c99da1e63f3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:08.764712 6289 raft_consensus.cc:1275] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 1 FOLLOWER]: Refusing update from remote peer 240f12b65a974b1ba74c99da1e63f3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:08.765332 6362 consensus_queue.cc:1048] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee432c4046a465a9d4d8f3666f95979" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:08.765488 6359 consensus_queue.cc:1048] T 0893b63071d444e9ac3872e9a11aa8bc P 240f12b65a974b1ba74c99da1e63f3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fadf05c13a34040a8f5d8c58fb1cbd2" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 43151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:08.771862 6365 mvcc.cc:204] Tried to move back new op lower bound from 7281055124533952512 to 7281055124434472960. Current Snapshot: MvccSnapshot[applied={T|T < 7281055124533952512}]
I20260501 02:09:08.773052 6367 mvcc.cc:204] Tried to move back new op lower bound from 7281055124533952512 to 7281055124434472960. Current Snapshot: MvccSnapshot[applied={T|T < 7281055124533952512}]
I20260501 02:09:08.774545 6366 mvcc.cc:204] Tried to move back new op lower bound from 7281055124533952512 to 7281055124434472960. Current Snapshot: MvccSnapshot[applied={T|T < 7281055124533952512 or (T in {7281055124533952512})}]
I20260501 02:09:08.781150 5885 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:08.781270 5885 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:08.783560 5885 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 5682eab723e84e59bdeed055c9adb8bd: Sending DeleteTablet for 3 replicas of tablet 0893b63071d444e9ac3872e9a11aa8bc
I20260501 02:09:08.784219 6138 tablet_service.cc:1558] Processing DeleteTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:08 UTC) from {username='slave'} at 127.0.0.1:35606
I20260501 02:09:08.784329 6007 tablet_service.cc:1558] Processing DeleteTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:08 UTC) from {username='slave'} at 127.0.0.1:32868
I20260501 02:09:08.784322 6269 tablet_service.cc:1558] Processing DeleteTablet for tablet 0893b63071d444e9ac3872e9a11aa8bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:08 UTC) from {username='slave'} at 127.0.0.1:34256
I20260501 02:09:08.784859 6376 tablet_replica.cc:333] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: stopping tablet replica
I20260501 02:09:08.785087 6376 raft_consensus.cc:2243] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:08.785336 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5945
I20260501 02:09:08.785378 6376 raft_consensus.cc:2272] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:08.786494 6375 tablet_replica.cc:333] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: stopping tablet replica
I20260501 02:09:08.786738 6375 raft_consensus.cc:2243] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:08.786988 6375 raft_consensus.cc:2272] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:08.788264 6375 ts_tablet_manager.cc:1916] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:08.788424 6376 ts_tablet_manager.cc:1916] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:08.790571 6375 ts_tablet_manager.cc:1929] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:08.790571 6376 ts_tablet_manager.cc:1929] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:08.790656 6376 log.cc:1199] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/0893b63071d444e9ac3872e9a11aa8bc
I20260501 02:09:08.790656 6375 log.cc:1199] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/0893b63071d444e9ac3872e9a11aa8bc
I20260501 02:09:08.790979 6375 ts_tablet_manager.cc:1950] T 0893b63071d444e9ac3872e9a11aa8bc P eee432c4046a465a9d4d8f3666f95979: Deleting consensus metadata
I20260501 02:09:08.790979 6376 ts_tablet_manager.cc:1950] T 0893b63071d444e9ac3872e9a11aa8bc P 7fadf05c13a34040a8f5d8c58fb1cbd2: Deleting consensus metadata
I20260501 02:09:08.792012 5872 catalog_manager.cc:5002] TS 7fadf05c13a34040a8f5d8c58fb1cbd2 (127.18.96.195:43151): tablet 0893b63071d444e9ac3872e9a11aa8bc (table test-table [id=690cf996564d4cedae111bc412ce759f]) successfully deleted
I20260501 02:09:08.792186 5870 catalog_manager.cc:5002] TS eee432c4046a465a9d4d8f3666f95979 (127.18.96.194:39763): tablet 0893b63071d444e9ac3872e9a11aa8bc (table test-table [id=690cf996564d4cedae111bc412ce759f]) successfully deleted
W20260501 02:09:08.792299 5872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:44013 (error 108)
I20260501 02:09:08.792598 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6076
I20260501 02:09:08.793570 6350 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:08.793256 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:44013 (local address 127.0.0.1:32882)
0501 02:09:08.793405 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:08.793474 (+ 69us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260501 02:09:08.793725 5872 catalog_manager.cc:4729] TS 240f12b65a974b1ba74c99da1e63f3e7 (127.18.96.193:44013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0893b63071d444e9ac3872e9a11aa8bc: Network error: Client connection negotiation failed: client connection to 127.18.96.193:44013: connect: Connection refused (error 111)
I20260501 02:09:08.798635 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6207
I20260501 02:09:08.804412 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 5854
2026-05-01T02:09:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (943 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:08 dist-test-slave-2j90 krb5kdc[6381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:08 dist-test-slave-2j90 krb5kdc[6381](info): set up 2 sockets
May 01 02:09:08 dist-test-slave-2j90 krb5kdc[6381](info): commencing operation
krb5kdc: starting...
W20260501 02:09:10.834443 18819 mini_kdc.cc:121] Time spent starting KDC: real 2.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:10 dist-test-slave-2j90 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:10.966864 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43125
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:11.065517 6397 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:11.065716 6397 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:11.065806 6397 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:11.069031 6397 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:11.069083 6397 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:11.069105 6397 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:11.069147 6397 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:11.069181 6397 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:11.073287 6397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:43125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:11.074301 6397 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:11.075078 6397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:11.080524 6403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.080639 6405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.080679 6402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:11.080830 6397 server_base.cc:1061] running on GCE node
I20260501 02:09:11.081283 6397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:11.082139 6397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:11.083285 6397 hybrid_clock.cc:648] HybridClock initialized: now 1777601351083266 us; error 32 us; skew 500 ppm
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:11.085856 6397 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:11.086863 6397 webserver.cc:492] Webserver started at http://127.18.96.254:41453/ using document root <none> and password file <none>
I20260501 02:09:11.087368 6397 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:11.087414 6397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:11.087558 6397 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:11.089089 6397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "90930bc9985a4997bb78e2ce7960a151"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "76c7c7d828651ebcd75a73ae631f2537"
server_key_iv: "235d288b06b7b232d38bc878cf5ed83c"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.089501 6397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "90930bc9985a4997bb78e2ce7960a151"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "76c7c7d828651ebcd75a73ae631f2537"
server_key_iv: "235d288b06b7b232d38bc878cf5ed83c"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.092509 6397 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260501 02:09:11.094417 6412 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.095283 6397 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:11.095412 6397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "90930bc9985a4997bb78e2ce7960a151"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "76c7c7d828651ebcd75a73ae631f2537"
server_key_iv: "235d288b06b7b232d38bc878cf5ed83c"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.095502 6397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:11.110855 6397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:11.113416 6397 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:11.113571 6397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:11.120321 6397 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:43125
I20260501 02:09:11.120342 6464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:43125 every 8 connection(s)
I20260501 02:09:11.121201 6397 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:11.121639 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6397
I20260501 02:09:11.121781 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:11.122087 18819 external_mini_cluster.cc:1468] Setting key 5cededf2024f3496fd70598449350f1d
I20260501 02:09:11.124035 6465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:11.130259 6465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: Bootstrap starting.
I20260501 02:09:11.132712 6465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:11.133494 6465 log.cc:826] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:11.135658 6465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: No bootstrap required, opened a new log
I20260501 02:09:11.136000 6468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.123252 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.18.96.254:43125)
0501 02:09:11.123642 (+ 390us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.123650 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.123678 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:09:11.124311 (+ 633us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.124329 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.124572 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.124813 (+ 241us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:11.125889 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.126694 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.127549 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.127841 (+ 292us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.130746 (+ 2905us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:11.130804 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:11.130817 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:11.130846 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:11.132920 (+ 2074us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.133433 (+ 513us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.133439 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.133444 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.133528 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.133825 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.133829 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.133830 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.134253 (+ 423us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:11.134426 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.135140 (+ 714us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.135360 (+ 220us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":128,"threads_started":1}
I20260501 02:09:11.138618 6465 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } }
I20260501 02:09:11.138774 6465 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:11.138808 6465 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90930bc9985a4997bb78e2ce7960a151, State: Initialized, Role: FOLLOWER
I20260501 02:09:11.139111 6465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } }
I20260501 02:09:11.139200 6465 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:11.139233 6465 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:11.139304 6465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:11.139995 6465 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } }
I20260501 02:09:11.140244 6465 leader_election.cc:304] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90930bc9985a4997bb78e2ce7960a151; no voters:
I20260501 02:09:11.140470 6465 leader_election.cc:290] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:11.140656 6470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:11.140847 6470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [term 1 LEADER]: Becoming Leader. State: Replica: 90930bc9985a4997bb78e2ce7960a151, State: Running, Role: LEADER
I20260501 02:09:11.141106 6470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } }
I20260501 02:09:11.141390 6465 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:11.142833 6471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90930bc9985a4997bb78e2ce7960a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } } }
I20260501 02:09:11.142963 6471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:11.143191 6472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90930bc9985a4997bb78e2ce7960a151. Latest consensus state: current_term: 1 leader_uuid: "90930bc9985a4997bb78e2ce7960a151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90930bc9985a4997bb78e2ce7960a151" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43125 } } }
I20260501 02:09:11.143272 6472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:11.143545 6479 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:11.146332 6479 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:11.150708 6479 catalog_manager.cc:1357] Generated new cluster ID: ee32e5201d5442d391cfd4e7017f84bd
I20260501 02:09:11.150771 6479 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:11.176241 6479 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:11.177088 6479 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:11.186280 6479 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: Generated new TSK 0
I20260501 02:09:11.186867 6479 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:11.245589 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:43125
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:11.345486 6493 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:11.345679 6493 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:11.345763 6493 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:11.349017 6493 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:11.349072 6493 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:11.349193 6493 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:11.353322 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:43125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:11.354362 6493 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:11.355139 6493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:11.361250 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.361331 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.361281 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:11.361481 6493 server_base.cc:1061] running on GCE node
I20260501 02:09:11.361847 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:11.362397 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:11.363544 6493 hybrid_clock.cc:648] HybridClock initialized: now 1777601351363529 us; error 30 us; skew 500 ppm
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:11.366145 6493 init.cc:377] Logged in from keytab as kudu/127.18.96.193@KRBTEST.COM (short username kudu)
I20260501 02:09:11.367206 6493 webserver.cc:492] Webserver started at http://127.18.96.193:42845/ using document root <none> and password file <none>
I20260501 02:09:11.367748 6493 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:11.367826 6493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:11.368033 6493 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:11.369678 6493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "9bdc7ba7787a47788fee2af7d2189f46"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "09c687a2550cd10b00b0484232c93346"
server_key_iv: "565c646c9da3d9b675f7914e9ad7910d"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.370246 6493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "9bdc7ba7787a47788fee2af7d2189f46"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "09c687a2550cd10b00b0484232c93346"
server_key_iv: "565c646c9da3d9b675f7914e9ad7910d"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.373497 6493 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260501 02:09:11.375699 6508 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.376642 6493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:11.376771 6493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "9bdc7ba7787a47788fee2af7d2189f46"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "09c687a2550cd10b00b0484232c93346"
server_key_iv: "565c646c9da3d9b675f7914e9ad7910d"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.376868 6493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:11.387375 6493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:11.390362 6493 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:11.390552 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:11.391076 6493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:11.391884 6493 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:11.391952 6493 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.392020 6493 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:11.392055 6493 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.401137 6493 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:35521
I20260501 02:09:11.401170 6621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:35521 every 8 connection(s)
I20260501 02:09:11.402020 6493 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.193@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:11.410578 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6493
I20260501 02:09:11.410755 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:11.411067 18819 external_mini_cluster.cc:1468] Setting key 23ecad887f26fb212a9a626818e3196c
I20260501 02:09:11.413945 6468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.403734 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:39671 (local address 127.18.96.254:43125)
0501 02:09:11.403882 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.403885 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.404564 (+ 679us) server_negotiation.cc:408] Connection header received
0501 02:09:11.405400 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.405404 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.405463 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.405565 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:11.407048 (+ 1483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.407555 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.408201 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.408398 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.410498 (+ 2100us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:11.410515 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:11.410517 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:11.410540 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:11.412054 (+ 1514us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.412662 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.412666 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.412668 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.412714 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.413013 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.413016 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.413018 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.413150 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:11.413249 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.413679 (+ 430us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.413791 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:09:11.414491 6624 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.403993 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:43125 (local address 127.18.96.193:39671)
0501 02:09:11.404424 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.404455 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.405200 (+ 745us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.405732 (+ 532us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.405740 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.406096 (+ 356us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:11.406895 (+ 799us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.406906 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.407676 (+ 770us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.407680 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.408068 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.408074 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.408272 (+ 198us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.408827 (+ 555us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:11.408846 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:11.410323 (+ 1477us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:11.412174 (+ 1851us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.412180 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.412190 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.412560 (+ 370us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.412806 (+ 246us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.412809 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.412811 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.412915 (+ 104us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.413248 (+ 333us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:11.413253 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:11.413454 (+ 201us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.413643 (+ 189us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.413840 (+ 197us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":143,"threads_started":1}
I20260501 02:09:11.415576 6622 heartbeater.cc:344] Connected to a master server at 127.18.96.254:43125
I20260501 02:09:11.415796 6622 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:11.416227 6622 heartbeater.cc:507] Master 127.18.96.254:43125 requested a full tablet report, sending...
I20260501 02:09:11.417740 6429 ts_manager.cc:194] Registered new tserver with Master: 9bdc7ba7787a47788fee2af7d2189f46 (127.18.96.193:35521)
I20260501 02:09:11.419528 6429 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.193@KRBTEST.COM'} at 127.18.96.193:39671
WARNING: no policy specified for kudu/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:11.457130 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:43125
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:11.553740 6629 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:11.553910 6629 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:11.554018 6629 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:11.557217 6629 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:11.557281 6629 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:11.557410 6629 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:11.561702 6629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:43125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:11.562726 6629 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:11.563498 6629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:11.569377 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.569435 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.569387 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:11.569697 6629 server_base.cc:1061] running on GCE node
I20260501 02:09:11.570089 6629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:11.570595 6629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:11.571775 6629 hybrid_clock.cc:648] HybridClock initialized: now 1777601351571763 us; error 35 us; skew 500 ppm
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:11.574261 6629 init.cc:377] Logged in from keytab as kudu/127.18.96.194@KRBTEST.COM (short username kudu)
I20260501 02:09:11.575256 6629 webserver.cc:492] Webserver started at http://127.18.96.194:39103/ using document root <none> and password file <none>
I20260501 02:09:11.575798 6629 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:11.575871 6629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:11.576073 6629 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:11.577675 6629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "7e15974b5df64599b25ba44c7b70dd1c"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "c82435b1e8232847efb0e7fb06a7287a"
server_key_iv: "a618640973fc5512f0cafda6adb4114a"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.578199 6629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "7e15974b5df64599b25ba44c7b70dd1c"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "c82435b1e8232847efb0e7fb06a7287a"
server_key_iv: "a618640973fc5512f0cafda6adb4114a"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.581197 6629 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:09:11.583225 6644 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.584101 6629 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:11.584227 6629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "7e15974b5df64599b25ba44c7b70dd1c"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "c82435b1e8232847efb0e7fb06a7287a"
server_key_iv: "a618640973fc5512f0cafda6adb4114a"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.584317 6629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:11.603591 6629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:11.606031 6629 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:11.606178 6629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:11.606652 6629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:11.607452 6629 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:11.607497 6629 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.607540 6629 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:11.607566 6629 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.616842 6629 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:35669
I20260501 02:09:11.616868 6757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:35669 every 8 connection(s)
I20260501 02:09:11.617760 6629 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:11.622200 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6629
I20260501 02:09:11.622347 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:11.622682 18819 external_mini_cluster.cc:1468] Setting key e20e1f9bc209026dc59acdd12c8d0250
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.194@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:11.630175 6468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.619559 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:49375 (local address 127.18.96.254:43125)
0501 02:09:11.619698 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.619701 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.620309 (+ 608us) server_negotiation.cc:408] Connection header received
0501 02:09:11.621137 (+ 828us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.621140 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.621207 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.621304 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:11.623120 (+ 1816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.623711 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.624442 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.624655 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.626628 (+ 1973us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:11.626649 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:11.626655 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:11.626690 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:11.628350 (+ 1660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.628800 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.628806 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.628810 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.628868 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.629141 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.629148 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.629153 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.629349 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:11.629455 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.629852 (+ 397us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.629990 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:11.630605 6760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.619816 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:43125 (local address 127.18.96.194:49375)
0501 02:09:11.620186 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.620217 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.620946 (+ 729us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.621454 (+ 508us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.621462 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.621790 (+ 328us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:11.622965 (+ 1175us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.622981 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.623871 (+ 890us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.623875 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.624315 (+ 440us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.624323 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.624472 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.625020 (+ 548us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:11.625038 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:11.626475 (+ 1437us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:11.628468 (+ 1993us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.628473 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.628481 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.628697 (+ 216us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.628966 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.628969 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.628970 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.629047 (+ 77us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.629455 (+ 408us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:11.629459 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:11.629627 (+ 168us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.629773 (+ 146us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.629926 (+ 153us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260501 02:09:11.631604 6758 heartbeater.cc:344] Connected to a master server at 127.18.96.254:43125
WARNING: no policy specified for kudu/127.18.96.195@KRBTEST.COM; defaulting to no policy
I20260501 02:09:11.631845 6758 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:11.632282 6758 heartbeater.cc:507] Master 127.18.96.254:43125 requested a full tablet report, sending...
I20260501 02:09:11.633199 6429 ts_manager.cc:194] Registered new tserver with Master: 7e15974b5df64599b25ba44c7b70dd1c (127.18.96.194:35669)
I20260501 02:09:11.633790 6429 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.194@KRBTEST.COM'} at 127.18.96.194:49375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:11.669155 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:43125
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:11.767700 6765 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:11.767887 6765 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:11.767977 6765 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:11.771252 6765 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:11.771307 6765 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:11.771433 6765 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:11.775480 6765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.18.96.254:43125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:11.776396 6765 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:11.777148 6765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:11.783097 6773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.783111 6770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:11.783118 6771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:11.783488 6765 server_base.cc:1061] running on GCE node
I20260501 02:09:11.783843 6765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:11.784411 6765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:11.785584 6765 hybrid_clock.cc:648] HybridClock initialized: now 1777601351785566 us; error 41 us; skew 500 ppm
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:11.788205 6765 init.cc:377] Logged in from keytab as kudu/127.18.96.195@KRBTEST.COM (short username kudu)
I20260501 02:09:11.789233 6765 webserver.cc:492] Webserver started at http://127.18.96.195:41819/ using document root <none> and password file <none>
I20260501 02:09:11.789707 6765 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:11.789753 6765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:11.789907 6765 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:11.791508 6765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "1675ab9a4435424c86d9948d020c7e50"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "3c75d7e59de7f7681a4d85826f2f216f"
server_key_iv: "4f67c100aca3d9f3b58325fa3a862cbb"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.791942 6765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "1675ab9a4435424c86d9948d020c7e50"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "3c75d7e59de7f7681a4d85826f2f216f"
server_key_iv: "4f67c100aca3d9f3b58325fa3a862cbb"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.795023 6765 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:09:11.796952 6780 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.797890 6765 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:11.798027 6765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "1675ab9a4435424c86d9948d020c7e50"
format_stamp: "Formatted at 2026-05-01 02:09:11 on dist-test-slave-2j90"
server_key: "3c75d7e59de7f7681a4d85826f2f216f"
server_key_iv: "4f67c100aca3d9f3b58325fa3a862cbb"
server_key_version: "encryptionkey@0"
I20260501 02:09:11.798103 6765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:11.815124 6765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:11.817641 6765 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:11.817780 6765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:11.818305 6765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:11.819092 6765 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:11.819140 6765 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.819180 6765 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:11.819196 6765 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:11.828445 6765 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:38435
I20260501 02:09:11.828518 6893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:38435 every 8 connection(s)
I20260501 02:09:11.829310 6765 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:09:11.834146 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6765
I20260501 02:09:11.834296 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:11.834558 18819 external_mini_cluster.cc:1468] Setting key 165ffdcfb7cddd423067afa845050b45
May 01 02:09:11 dist-test-slave-2j90 krb5kdc[6381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.195@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:11.841061 6468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.830998 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:44351 (local address 127.18.96.254:43125)
0501 02:09:11.831132 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.831136 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.831721 (+ 585us) server_negotiation.cc:408] Connection header received
0501 02:09:11.832542 (+ 821us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.832546 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.832625 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.832720 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:11.834313 (+ 1593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.834950 (+ 637us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.835794 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.836009 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.838187 (+ 2178us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:11.838200 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:11.838202 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:11.838226 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:11.839454 (+ 1228us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.839877 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.839880 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.839881 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.839921 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.840159 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.840161 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.840162 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.840281 (+ 119us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:11.840352 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.840859 (+ 507us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.840951 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:09:11.841470 6896 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.831228 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:43125 (local address 127.18.96.195:44351)
0501 02:09:11.831597 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.831627 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.832357 (+ 730us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.832860 (+ 503us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.832868 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.833192 (+ 324us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:11.834142 (+ 950us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.834160 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.835095 (+ 935us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.835099 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.835674 (+ 575us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.835683 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.835862 (+ 179us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.836456 (+ 594us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:11.836482 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:11.838061 (+ 1579us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:11.839574 (+ 1513us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.839579 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.839586 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.839792 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.839996 (+ 204us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:11.839998 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:11.840000 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:11.840088 (+ 88us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:11.840371 (+ 283us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:11.840375 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:11.840535 (+ 160us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.840690 (+ 155us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.840855 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260501 02:09:11.842486 6894 heartbeater.cc:344] Connected to a master server at 127.18.96.254:43125
I20260501 02:09:11.842707 6894 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:11.843132 6894 heartbeater.cc:507] Master 127.18.96.254:43125 requested a full tablet report, sending...
I20260501 02:09:11.844022 6429 ts_manager.cc:194] Registered new tserver with Master: 1675ab9a4435424c86d9948d020c7e50 (127.18.96.195:38435)
I20260501 02:09:11.844575 6429 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.96.195@KRBTEST.COM'} at 127.18.96.195:44351
I20260501 02:09:11.848119 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:11.856209 6468 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.849833 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.18.96.254:43125)
0501 02:09:11.850085 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.850089 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.850231 (+ 142us) server_negotiation.cc:408] Connection header received
0501 02:09:11.850384 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.850388 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.850448 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.850535 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:11.851269 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.851750 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.852367 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.852557 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.853465 (+ 908us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:11.853478 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:11.853481 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:11.853502 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:11.854889 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.855210 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.855213 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.855214 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.855249 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:11.855504 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:11.855507 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:11.855508 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:11.855649 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:11.855756 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.855919 (+ 163us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.856002 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260501 02:09:11.858553 6429 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:11.860811 6429 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:11.873363 6905 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.868552 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38435 (local address 127.0.0.1:41130)
0501 02:09:11.868836 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.868866 (+ 30us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.868980 (+ 114us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.870355 (+ 1375us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.870358 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.870376 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:11.870646 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.870652 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.872064 (+ 1412us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.872067 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.872898 (+ 831us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.872907 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.873021 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.873068 (+ 47us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.873152 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.873207 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260501 02:09:11.873798 6909 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.869405 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35521 (local address 127.0.0.1:44652)
0501 02:09:11.869696 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.869710 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.869779 (+ 69us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.870604 (+ 825us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.870607 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.870624 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:11.870834 (+ 210us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.870839 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.871799 (+ 960us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.871805 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.872901 (+ 1096us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.872907 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.873019 (+ 112us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.873068 (+ 49us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.873152 (+ 84us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.873207 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260501 02:09:11.873934 6906 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.868656 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.18.96.195:38435)
0501 02:09:11.868948 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.868954 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.868975 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:09:11.870006 (+ 1031us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.870011 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.870205 (+ 194us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.870358 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:11.870763 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.871948 (+ 1185us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.873041 (+ 1093us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.873568 (+ 527us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.873665 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.873724 (+ 59us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.873798 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":131,"threads_started":1}
I20260501 02:09:11.873799 6907 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.869162 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35669 (local address 127.0.0.1:45242)
0501 02:09:11.869407 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:11.869438 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:11.869523 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:11.870154 (+ 631us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:11.870159 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:11.870182 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:11.870478 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.870486 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.872093 (+ 1607us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.872096 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:11.872938 (+ 842us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:11.872946 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.873055 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.873069 (+ 14us) client_negotiation.cc:770] Sending connection context
0501 02:09:11.873140 (+ 71us) client_negotiation.cc:241] Negotiation successful
0501 02:09:11.873209 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260501 02:09:11.874197 6908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.869307 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.18.96.194:35669)
0501 02:09:11.869649 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.869655 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.869676 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:09:11.869759 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.869764 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.869935 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.870626 (+ 691us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:11.870753 (+ 127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.871948 (+ 1195us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.873340 (+ 1392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.873817 (+ 477us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.873898 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.873984 (+ 86us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.874067 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":167,"threads_started":1}
I20260501 02:09:11.875595 6910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:11.869803 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.18.96.193:35521)
0501 02:09:11.870211 (+ 408us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:11.870217 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:11.870236 (+ 19us) server_negotiation.cc:408] Connection header received
0501 02:09:11.870310 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:11.870314 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:11.870462 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:11.870603 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:11.870952 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.871682 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:11.874706 (+ 3024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:11.875205 (+ 499us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:11.875289 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:11.875386 (+ 97us) server_negotiation.cc:300] Negotiation successful
0501 02:09:11.875465 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":125,"threads_started":1}
I20260501 02:09:11.876607 6692 tablet_service.cc:1511] Processing CreateTablet for tablet b10002cfb5bd406a9ff282e325de2d19 (DEFAULT_TABLE table=test-table [id=b1619bcb699749cd8a5a718f7037575d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:11.876623 6828 tablet_service.cc:1511] Processing CreateTablet for tablet b10002cfb5bd406a9ff282e325de2d19 (DEFAULT_TABLE table=test-table [id=b1619bcb699749cd8a5a718f7037575d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:11.877350 6692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10002cfb5bd406a9ff282e325de2d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:11.877455 6828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10002cfb5bd406a9ff282e325de2d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:11.877801 6556 tablet_service.cc:1511] Processing CreateTablet for tablet b10002cfb5bd406a9ff282e325de2d19 (DEFAULT_TABLE table=test-table [id=b1619bcb699749cd8a5a718f7037575d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:11.878605 6556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10002cfb5bd406a9ff282e325de2d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:11.882761 6911 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Bootstrap starting.
I20260501 02:09:11.883534 6912 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Bootstrap starting.
I20260501 02:09:11.883966 6913 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Bootstrap starting.
I20260501 02:09:11.884943 6911 tablet_bootstrap.cc:654] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:11.885360 6912 tablet_bootstrap.cc:654] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:11.885741 6911 log.cc:826] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:11.885936 6913 tablet_bootstrap.cc:654] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:11.886210 6912 log.cc:826] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:11.886709 6913 log.cc:826] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:11.887696 6911 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: No bootstrap required, opened a new log
I20260501 02:09:11.887894 6911 ts_tablet_manager.cc:1403] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:09:11.888008 6912 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: No bootstrap required, opened a new log
I20260501 02:09:11.888185 6912 ts_tablet_manager.cc:1403] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:09:11.888325 6913 tablet_bootstrap.cc:492] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: No bootstrap required, opened a new log
I20260501 02:09:11.888507 6913 ts_tablet_manager.cc:1403] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260501 02:09:11.890914 6911 raft_consensus.cc:359] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.891106 6911 raft_consensus.cc:385] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:11.891171 6911 raft_consensus.cc:740] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e15974b5df64599b25ba44c7b70dd1c, State: Initialized, Role: FOLLOWER
I20260501 02:09:11.891583 6911 consensus_queue.cc:260] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.891831 6912 raft_consensus.cc:359] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.892019 6912 raft_consensus.cc:385] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:11.892076 6912 raft_consensus.cc:740] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1675ab9a4435424c86d9948d020c7e50, State: Initialized, Role: FOLLOWER
I20260501 02:09:11.892309 6758 heartbeater.cc:499] Master 127.18.96.254:43125 was elected leader, sending a full tablet report...
I20260501 02:09:11.892436 6911 ts_tablet_manager.cc:1434] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:09:11.892454 6912 consensus_queue.cc:260] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.892899 6913 raft_consensus.cc:359] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.893065 6894 heartbeater.cc:499] Master 127.18.96.254:43125 was elected leader, sending a full tablet report...
I20260501 02:09:11.893078 6913 raft_consensus.cc:385] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:11.893124 6913 raft_consensus.cc:740] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bdc7ba7787a47788fee2af7d2189f46, State: Initialized, Role: FOLLOWER
I20260501 02:09:11.893296 6912 ts_tablet_manager.cc:1434] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260501 02:09:11.893472 6913 consensus_queue.cc:260] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:11.894117 6622 heartbeater.cc:499] Master 127.18.96.254:43125 was elected leader, sending a full tablet report...
I20260501 02:09:11.894284 6913 ts_tablet_manager.cc:1434] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260501 02:09:11.903008 6623 tablet.cc:2404] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:12.080209 6895 tablet.cc:2404] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:12.081040 6917 raft_consensus.cc:493] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:12.081218 6917 raft_consensus.cc:515] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:12.082274 6917 leader_election.cc:290] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1675ab9a4435424c86d9948d020c7e50 (127.18.96.195:38435), 9bdc7ba7787a47788fee2af7d2189f46 (127.18.96.193:35521)
I20260501 02:09:12.085005 6918 raft_consensus.cc:493] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:12.085153 6918 raft_consensus.cc:515] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:12.085671 6760 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.082529 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:38435 (local address 127.18.96.194:41711)
0501 02:09:12.082659 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.082675 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.082769 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.083130 (+ 361us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.083133 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.083146 (+ 13us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.083416 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.083424 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.084392 (+ 968us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.084395 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.085255 (+ 860us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.085263 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.085404 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.085427 (+ 23us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.085487 (+ 60us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.085545 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260501 02:09:12.086124 6906 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.082705 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:41711 (local address 127.18.96.195:38435)
0501 02:09:12.082858 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.082862 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.082889 (+ 27us) server_negotiation.cc:408] Connection header received
0501 02:09:12.082945 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.082950 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.083009 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.083080 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.083546 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.084265 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.085373 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.085865 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.085897 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.085945 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.086024 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260501 02:09:12.086529 6910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.082608 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:46977 (local address 127.18.96.193:35521)
0501 02:09:12.082748 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.082755 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.083137 (+ 382us) server_negotiation.cc:408] Connection header received
0501 02:09:12.083227 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.083229 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.083269 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.083328 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.083736 (+ 408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.084466 (+ 730us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.085522 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.086087 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.086116 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.086339 (+ 223us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.086405 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:09:12.086655 6848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "7e15974b5df64599b25ba44c7b70dd1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1675ab9a4435424c86d9948d020c7e50" is_pre_election: true
I20260501 02:09:12.086834 6848 raft_consensus.cc:2468] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e15974b5df64599b25ba44c7b70dd1c in term 0.
I20260501 02:09:12.086877 6920 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.082529 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35521 (local address 127.18.96.194:46977)
0501 02:09:12.083022 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.083032 (+ 10us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.083143 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.083367 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.083370 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.083386 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.083635 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.083641 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.084577 (+ 936us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.084580 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.085385 (+ 805us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.085396 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.086167 (+ 771us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.086178 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.086669 (+ 491us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.086732 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":133,"thread_start_us":130,"threads_started":1}
I20260501 02:09:12.087234 6918 leader_election.cc:290] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e15974b5df64599b25ba44c7b70dd1c (127.18.96.194:35669), 9bdc7ba7787a47788fee2af7d2189f46 (127.18.96.193:35521)
I20260501 02:09:12.087283 6645 leader_election.cc:304] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1675ab9a4435424c86d9948d020c7e50, 7e15974b5df64599b25ba44c7b70dd1c; no voters:
I20260501 02:09:12.087472 6576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "7e15974b5df64599b25ba44c7b70dd1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bdc7ba7787a47788fee2af7d2189f46" is_pre_election: true
I20260501 02:09:12.087559 6917 raft_consensus.cc:2804] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:12.087637 6917 raft_consensus.cc:493] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:12.087711 6917 raft_consensus.cc:3060] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:12.087759 6576 raft_consensus.cc:2468] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e15974b5df64599b25ba44c7b70dd1c in term 0.
I20260501 02:09:12.089018 6917 raft_consensus.cc:515] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:12.089444 6896 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.086473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35669 (local address 127.18.96.195:48371)
0501 02:09:12.086654 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.086668 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.086748 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.087274 (+ 526us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.087278 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.087302 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.087515 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.087522 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.088493 (+ 971us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.088496 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.089129 (+ 633us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.089138 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.089231 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.089242 (+ 11us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.089285 (+ 43us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.089331 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260501 02:09:12.089641 6848 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "7e15974b5df64599b25ba44c7b70dd1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1675ab9a4435424c86d9948d020c7e50"
I20260501 02:09:12.089743 6848 raft_consensus.cc:3060] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:12.089898 6908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.086616 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:48371 (local address 127.18.96.194:35669)
0501 02:09:12.086961 (+ 345us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.086966 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.086982 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:09:12.087019 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.087022 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.087060 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.087133 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.087664 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.088353 (+ 689us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.089253 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.089680 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.089710 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.089756 (+ 46us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.089806 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260501 02:09:12.090298 6712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "1675ab9a4435424c86d9948d020c7e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e15974b5df64599b25ba44c7b70dd1c" is_pre_election: true
I20260501 02:09:12.090353 6917 leader_election.cc:290] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [CANDIDATE]: Term 1 election: Requested vote from peers 1675ab9a4435424c86d9948d020c7e50 (127.18.96.195:38435), 9bdc7ba7787a47788fee2af7d2189f46 (127.18.96.193:35521)
I20260501 02:09:12.090492 6712 raft_consensus.cc:2393] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1675ab9a4435424c86d9948d020c7e50 in current term 1: Already voted for candidate 7e15974b5df64599b25ba44c7b70dd1c in this term.
I20260501 02:09:12.090970 6848 raft_consensus.cc:2468] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e15974b5df64599b25ba44c7b70dd1c in term 1.
I20260501 02:09:12.091089 6576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "7e15974b5df64599b25ba44c7b70dd1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bdc7ba7787a47788fee2af7d2189f46"
I20260501 02:09:12.091199 6576 raft_consensus.cc:3060] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:12.091221 6645 leader_election.cc:304] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1675ab9a4435424c86d9948d020c7e50, 7e15974b5df64599b25ba44c7b70dd1c; no voters:
I20260501 02:09:12.091364 6917 raft_consensus.cc:2804] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:12.091616 6910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.087933 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:59635 (local address 127.18.96.193:35521)
0501 02:09:12.088060 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.088064 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.088415 (+ 351us) server_negotiation.cc:408] Connection header received
0501 02:09:12.088507 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.088510 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.088553 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.088622 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.089091 (+ 469us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.089703 (+ 612us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.090753 (+ 1050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.091164 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.091192 (+ 28us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.091446 (+ 254us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.091515 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260501 02:09:12.091610 6917 raft_consensus.cc:697] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 LEADER]: Becoming Leader. State: Replica: 7e15974b5df64599b25ba44c7b70dd1c, State: Running, Role: LEADER
I20260501 02:09:12.091921 6921 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.087556 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35521 (local address 127.18.96.195:59635)
0501 02:09:12.088318 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.088331 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.088430 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.088654 (+ 224us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.088660 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.088684 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.088966 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.088976 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.089806 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.089811 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.090616 (+ 805us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.090624 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.091250 (+ 626us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.091265 (+ 15us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.091733 (+ 468us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.091785 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":141,"threads_started":1}
I20260501 02:09:12.091957 6917 consensus_queue.cc:237] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } }
I20260501 02:09:12.092366 6575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10002cfb5bd406a9ff282e325de2d19" candidate_uuid: "1675ab9a4435424c86d9948d020c7e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bdc7ba7787a47788fee2af7d2189f46" is_pre_election: true
I20260501 02:09:12.092455 6576 raft_consensus.cc:2468] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e15974b5df64599b25ba44c7b70dd1c in term 1.
I20260501 02:09:12.092813 6783 leader_election.cc:304] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1675ab9a4435424c86d9948d020c7e50; no voters: 7e15974b5df64599b25ba44c7b70dd1c, 9bdc7ba7787a47788fee2af7d2189f46
I20260501 02:09:12.093178 6918 raft_consensus.cc:2749] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260501 02:09:12.095575 6427 catalog_manager.cc:5671] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e15974b5df64599b25ba44c7b70dd1c (127.18.96.194). New cstate: current_term: 1 leader_uuid: "7e15974b5df64599b25ba44c7b70dd1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e15974b5df64599b25ba44c7b70dd1c" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 35669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 } health_report { overall_health: UNKNOWN } } }
I20260501 02:09:12.107429 6908 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.104687 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.18.96.194:35669)
0501 02:09:12.104810 (+ 123us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.104816 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.104849 (+ 33us) server_negotiation.cc:408] Connection header received
0501 02:09:12.105031 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.105034 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.105078 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.105170 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:12.105511 (+ 341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.105988 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.106645 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.106775 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.106835 (+ 60us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:12.107111 (+ 276us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:12.107187 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.107262 (+ 75us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.107298 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:09:12.112000 6575 raft_consensus.cc:1275] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 1 FOLLOWER]: Refusing update from remote peer 7e15974b5df64599b25ba44c7b70dd1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:12.112027 6848 raft_consensus.cc:1275] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 1 FOLLOWER]: Refusing update from remote peer 7e15974b5df64599b25ba44c7b70dd1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:12.112586 6922 consensus_queue.cc:1048] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1675ab9a4435424c86d9948d020c7e50" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:12.112759 6917 consensus_queue.cc:1048] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bdc7ba7787a47788fee2af7d2189f46" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:12.121631 6926 mvcc.cc:204] Tried to move back new op lower bound from 7281055138244943872 to 7281055138173472768. Current Snapshot: MvccSnapshot[applied={T|T < 7281055138244943872}]
I20260501 02:09:12.128737 6428 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:12.128885 6428 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:12.131055 6428 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 90930bc9985a4997bb78e2ce7960a151: Sending DeleteTablet for 3 replicas of tablet b10002cfb5bd406a9ff282e325de2d19
I20260501 02:09:12.131906 6828 tablet_service.cc:1558] Processing DeleteTablet for tablet b10002cfb5bd406a9ff282e325de2d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:12 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:41130
I20260501 02:09:12.131902 6692 tablet_service.cc:1558] Processing DeleteTablet for tablet b10002cfb5bd406a9ff282e325de2d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:12 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:45242
I20260501 02:09:12.132345 6556 tablet_service.cc:1558] Processing DeleteTablet for tablet b10002cfb5bd406a9ff282e325de2d19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:12 UTC) from {username='kudu', principal='kudu/127.18.96.254@KRBTEST.COM'} at 127.0.0.1:44652
I20260501 02:09:12.132763 6937 tablet_replica.cc:333] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46: stopping tablet replica
I20260501 02:09:12.133040 6937 raft_consensus.cc:2243] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:12.133364 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6493
I20260501 02:09:12.133358 6937 raft_consensus.cc:2272] T b10002cfb5bd406a9ff282e325de2d19 P 9bdc7ba7787a47788fee2af7d2189f46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:12.139416 6936 tablet_replica.cc:333] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: stopping tablet replica
I20260501 02:09:12.139685 6935 tablet_replica.cc:333] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: stopping tablet replica
I20260501 02:09:12.139674 6936 raft_consensus.cc:2243] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:12.139938 6936 raft_consensus.cc:2272] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:12.139919 6935 raft_consensus.cc:2243] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:09:12.140508 6935 raft_consensus.cc:2272] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:12.141340 6936 ts_tablet_manager.cc:1916] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:12.141791 6935 ts_tablet_manager.cc:1916] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260501 02:09:12.143430 6415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:35521 (error 108)
I20260501 02:09:12.144028 6936 ts_tablet_manager.cc:1929] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:12.144119 6936 log.cc:1199] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/b10002cfb5bd406a9ff282e325de2d19
I20260501 02:09:12.144496 6936 ts_tablet_manager.cc:1950] T b10002cfb5bd406a9ff282e325de2d19 P 1675ab9a4435424c86d9948d020c7e50: Deleting consensus metadata
I20260501 02:09:12.144814 6907 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.144513 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:35521 (local address 127.0.0.1:44654)
0501 02:09:12.144687 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.144740 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:35521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260501 02:09:12.144845 6935 ts_tablet_manager.cc:1929] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260501 02:09:12.144958 6415 catalog_manager.cc:4729] TS 9bdc7ba7787a47788fee2af7d2189f46 (127.18.96.193:35521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b10002cfb5bd406a9ff282e325de2d19: Network error: Client connection negotiation failed: client connection to 127.18.96.193:35521: connect: Connection refused (error 111)
I20260501 02:09:12.145088 6935 log.cc:1199] T b10002cfb5bd406a9ff282e325de2d19 P 7e15974b5df64599b25ba44c7b70dd1c: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/b10002cfb5bd406a9ff282e325de2d19
I20260501 02:09:12.145205 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6629
I20260501 02:09:12.145608 6413 catalog_manager.cc:5002] TS 1675ab9a4435424c86d9948d020c7e50 (127.18.96.195:38435): tablet b10002cfb5bd406a9ff282e325de2d19 (table test-table [id=b1619bcb699749cd8a5a718f7037575d]) successfully deleted
I20260501 02:09:12.151520 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6765
I20260501 02:09:12.152474 6907 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.151620 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:35669 (local address 127.0.0.1:45254)
0501 02:09:12.152365 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.152404 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.194:35669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643}
I20260501 02:09:12.156963 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6397
2026-05-01T02:09:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3353 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-05-01T02:09:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:12Z Disabled control of system clock
I20260501 02:09:12.185819 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40433
--webserver_interface=127.18.96.254
--webserver_port=0
--builtin_ntp_servers=127.18.96.212:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:40433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:12.283095 6942 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:12.283277 6942 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:12.283356 6942 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:12.286494 6942 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:12.286540 6942 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:12.286556 6942 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:12.286574 6942 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:12.286590 6942 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:12.290081 6942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:40433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:12.291020 6942 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:12.292055 6942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:12.297271 6950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.297401 6947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.297277 6948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.297652 6942 server_base.cc:1061] running on GCE node
I20260501 02:09:12.298125 6942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:12.298930 6942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:12.300128 6942 hybrid_clock.cc:648] HybridClock initialized: now 1777601352300104 us; error 36 us; skew 500 ppm
I20260501 02:09:12.301841 6942 webserver.cc:492] Webserver started at http://127.18.96.254:44429/ using document root <none> and password file <none>
I20260501 02:09:12.302421 6942 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:12.302505 6942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:12.302726 6942 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:12.304296 6942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "283d6cff9ab94729902347b615fd20c9"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "48ac293ffc3e0ed9af59924e1187d4da"
server_key_iv: "a29424860412ad0cdeab1486cc737dec"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.304773 6942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "283d6cff9ab94729902347b615fd20c9"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "48ac293ffc3e0ed9af59924e1187d4da"
server_key_iv: "a29424860412ad0cdeab1486cc737dec"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.307767 6942 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:12.309638 6956 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.310573 6942 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:12.310700 6942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "283d6cff9ab94729902347b615fd20c9"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "48ac293ffc3e0ed9af59924e1187d4da"
server_key_iv: "a29424860412ad0cdeab1486cc737dec"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.310788 6942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:12.328917 6942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:12.329496 6942 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:12.329681 6942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:12.336455 6942 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:40433
I20260501 02:09:12.336483 7008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:40433 every 8 connection(s)
I20260501 02:09:12.337347 6942 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:12.339836 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:12.340204 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 6942
I20260501 02:09:12.340374 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:12.340627 18819 external_mini_cluster.cc:1468] Setting key 62860315d61424f38573b8643badfef0
I20260501 02:09:12.345119 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: Bootstrap starting.
I20260501 02:09:12.347502 7009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:12.347682 7012 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.341719 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.18.96.254:40433)
0501 02:09:12.342264 (+ 545us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.342273 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.342324 (+ 51us) server_negotiation.cc:408] Connection header received
0501 02:09:12.342827 (+ 503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.342840 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.343118 (+ 278us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.343416 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:12.343912 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.344653 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.345366 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.345629 (+ 263us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.346016 (+ 387us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:12.346070 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:12.346083 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:12.346510 (+ 427us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:12.346547 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:12.346562 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:12.346669 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.346902 (+ 233us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.347025 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20260501 02:09:12.348263 7009 log.cc:826] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:12.349928 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: No bootstrap required, opened a new log
I20260501 02:09:12.352089 7009 raft_consensus.cc:359] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } }
I20260501 02:09:12.352242 7009 raft_consensus.cc:385] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:12.352277 7009 raft_consensus.cc:740] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283d6cff9ab94729902347b615fd20c9, State: Initialized, Role: FOLLOWER
I20260501 02:09:12.352566 7009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } }
I20260501 02:09:12.352667 7009 raft_consensus.cc:399] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:12.352708 7009 raft_consensus.cc:493] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:12.352768 7009 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:12.353433 7009 raft_consensus.cc:515] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } }
I20260501 02:09:12.353693 7009 leader_election.cc:304] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 283d6cff9ab94729902347b615fd20c9; no voters:
I20260501 02:09:12.353900 7009 leader_election.cc:290] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:12.354038 7014 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:12.354252 7014 raft_consensus.cc:697] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [term 1 LEADER]: Becoming Leader. State: Replica: 283d6cff9ab94729902347b615fd20c9, State: Running, Role: LEADER
I20260501 02:09:12.354525 7014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } }
I20260501 02:09:12.354964 7009 sys_catalog.cc:565] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:12.356295 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 283d6cff9ab94729902347b615fd20c9. Latest consensus state: current_term: 1 leader_uuid: "283d6cff9ab94729902347b615fd20c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } } }
I20260501 02:09:12.356456 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:12.356295 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "283d6cff9ab94729902347b615fd20c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "283d6cff9ab94729902347b615fd20c9" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 40433 } } }
I20260501 02:09:12.356707 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:12.356932 7023 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:12.359478 7023 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:12.364049 7023 catalog_manager.cc:1357] Generated new cluster ID: 8f05191b5da54c1d92c8714bb692c8e5
I20260501 02:09:12.364111 7023 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:12.378429 7023 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:12.379463 7023 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:12.396420 7023 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: Generated new TSK 0
I20260501 02:09:12.397022 7023 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:12.405638 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.193:0
--local_ip_for_outbound_sockets=127.18.96.193
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--builtin_ntp_servers=127.18.96.212:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:12.503430 7033 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:12.503647 7033 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:12.503729 7033 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:12.506907 7033 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:12.506976 7033 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:12.507069 7033 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.193
I20260501 02:09:12.510566 7033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.96.193
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.193
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:12.511466 7033 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:12.512486 7033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:12.518457 7039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.518409 7041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.518412 7038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.518680 7033 server_base.cc:1061] running on GCE node
I20260501 02:09:12.519054 7033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:12.519556 7033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:12.520711 7033 hybrid_clock.cc:648] HybridClock initialized: now 1777601352520677 us; error 40 us; skew 500 ppm
I20260501 02:09:12.522389 7033 webserver.cc:492] Webserver started at http://127.18.96.193:36365/ using document root <none> and password file <none>
I20260501 02:09:12.522904 7033 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:12.522956 7033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:12.523187 7033 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:12.524745 7033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data/instance:
uuid: "adf8faebff79470aa3c26b0f539267fc"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "aa6e11c13578098f2252a5912df3b473"
server_key_iv: "75d67b8df360e0e1a62acbb467323fc6"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.525203 7033 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance:
uuid: "adf8faebff79470aa3c26b0f539267fc"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "aa6e11c13578098f2252a5912df3b473"
server_key_iv: "75d67b8df360e0e1a62acbb467323fc6"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.528165 7033 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:09:12.530092 7047 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.530890 7033 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:12.531013 7033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal
uuid: "adf8faebff79470aa3c26b0f539267fc"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "aa6e11c13578098f2252a5912df3b473"
server_key_iv: "75d67b8df360e0e1a62acbb467323fc6"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.531113 7033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:12.559152 7033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:12.559759 7033 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:12.559942 7033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:12.560506 7033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:12.561453 7033 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:12.561525 7033 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.561595 7033 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:12.561647 7033 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.570559 7033 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.193:38245
I20260501 02:09:12.570662 7160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.193:38245 every 8 connection(s)
I20260501 02:09:12.571436 7033 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/data/info.pb
I20260501 02:09:12.578070 7012 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.573093 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.193:46073 (local address 127.18.96.254:40433)
0501 02:09:12.573244 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.573249 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.574093 (+ 844us) server_negotiation.cc:408] Connection header received
0501 02:09:12.574903 (+ 810us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.574905 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.574955 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.575027 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:12.575749 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.576243 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.576825 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.576961 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.577620 (+ 659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:12.577635 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:12.577637 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:12.577647 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:12.577662 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:12.577668 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:12.577725 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.577859 (+ 134us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.577903 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:12.578614 7163 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.573392 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40433 (local address 127.18.96.193:46073)
0501 02:09:12.573920 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.573993 (+ 73us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.574743 (+ 750us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.575165 (+ 422us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.575171 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.575223 (+ 52us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:12.575629 (+ 406us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.575638 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.576348 (+ 710us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.576351 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.576730 (+ 379us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.576736 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.576873 (+ 137us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.577446 (+ 573us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:12.577447 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:12.577469 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:12.577509 (+ 40us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:12.577743 (+ 234us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:12.577748 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:12.577751 (+ 3us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.577846 (+ 95us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.577991 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
I20260501 02:09:12.579650 7161 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40433
I20260501 02:09:12.579854 7161 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:12.580268 7161 heartbeater.cc:507] Master 127.18.96.254:40433 requested a full tablet report, sending...
I20260501 02:09:12.580914 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7033
I20260501 02:09:12.581053 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-0/wal/instance
I20260501 02:09:12.581301 18819 external_mini_cluster.cc:1468] Setting key 80443beb1f5223a508788fbb07d99e59
I20260501 02:09:12.581617 6973 ts_manager.cc:194] Registered new tserver with Master: adf8faebff79470aa3c26b0f539267fc (127.18.96.193:38245)
I20260501 02:09:12.582893 6973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.193:46073
I20260501 02:09:12.583290 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.194:0
--local_ip_for_outbound_sockets=127.18.96.194
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--builtin_ntp_servers=127.18.96.212:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260501 02:09:12.677125 7164 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:12.677287 7164 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:12.677356 7164 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:12.680512 7164 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:12.680572 7164 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:12.680682 7164 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.194
I20260501 02:09:12.684281 7164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.96.194
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.194
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:12.685124 7164 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:12.686182 7164 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:12.692044 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.692178 7164 server_base.cc:1061] running on GCE node
W20260501 02:09:12.692004 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.691998 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.692706 7164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:12.693216 7164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:12.694386 7164 hybrid_clock.cc:648] HybridClock initialized: now 1777601352694368 us; error 39 us; skew 500 ppm
I20260501 02:09:12.695923 7164 webserver.cc:492] Webserver started at http://127.18.96.194:35127/ using document root <none> and password file <none>
I20260501 02:09:12.696440 7164 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:12.696522 7164 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:12.696743 7164 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:12.698310 7164 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data/instance:
uuid: "8c5c1a0f71024a7083e575a09b543e04"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "687fa24a3f88504da4f93a5bf1fb06fd"
server_key_iv: "b373c7ae8621c25ad396fed85882a109"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.698770 7164 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance:
uuid: "8c5c1a0f71024a7083e575a09b543e04"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "687fa24a3f88504da4f93a5bf1fb06fd"
server_key_iv: "b373c7ae8621c25ad396fed85882a109"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.701618 7164 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:12.703603 7178 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.704468 7164 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:12.704596 7164 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal
uuid: "8c5c1a0f71024a7083e575a09b543e04"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "687fa24a3f88504da4f93a5bf1fb06fd"
server_key_iv: "b373c7ae8621c25ad396fed85882a109"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.704687 7164 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:12.719400 7164 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:12.719929 7164 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:12.720101 7164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:12.720628 7164 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:12.721514 7164 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:12.721586 7164 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.721653 7164 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:12.721701 7164 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.730337 7164 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.194:42771
I20260501 02:09:12.730373 7291 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.194:42771 every 8 connection(s)
I20260501 02:09:12.731122 7164 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/data/info.pb
I20260501 02:09:12.737521 7012 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.732680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.194:33089 (local address 127.18.96.254:40433)
0501 02:09:12.732808 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.732811 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.733499 (+ 688us) server_negotiation.cc:408] Connection header received
0501 02:09:12.734305 (+ 806us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.734308 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.734372 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.734460 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:12.735246 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.735795 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.736367 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.736500 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.737126 (+ 626us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:12.737143 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:12.737146 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:12.737155 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:12.737169 (+ 14us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:12.737176 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:12.737233 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.737372 (+ 139us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.737415 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:09:12.738083 7294 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.732854 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40433 (local address 127.18.96.194:33089)
0501 02:09:12.733368 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.733395 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.734137 (+ 742us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.734612 (+ 475us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.734619 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.734671 (+ 52us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:12.735116 (+ 445us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.735126 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.735892 (+ 766us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.735894 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.736267 (+ 373us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.736273 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.736426 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.736963 (+ 537us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:12.736964 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:12.736985 (+ 21us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:12.737024 (+ 39us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:12.737247 (+ 223us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:12.737252 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:12.737256 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.737356 (+ 100us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.737454 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260501 02:09:12.738821 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7164
I20260501 02:09:12.738968 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal/instance
I20260501 02:09:12.739107 7292 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40433
I20260501 02:09:12.739281 18819 external_mini_cluster.cc:1468] Setting key 4255886015a27a678ed31071dbd12cd7
I20260501 02:09:12.739343 7292 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:12.739867 7292 heartbeater.cc:507] Master 127.18.96.254:40433 requested a full tablet report, sending...
I20260501 02:09:12.740839 6973 ts_manager.cc:194] Registered new tserver with Master: 8c5c1a0f71024a7083e575a09b543e04 (127.18.96.194:42771)
I20260501 02:09:12.741343 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.96.195:0
--local_ip_for_outbound_sockets=127.18.96.195
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--builtin_ntp_servers=127.18.96.212:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260501 02:09:12.741380 6973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.194:33089
W20260501 02:09:12.839576 7295 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:12.839792 7295 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:12.839871 7295 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:12.843039 7295 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:12.843120 7295 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:12.843222 7295 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.96.195
I20260501 02:09:12.846869 7295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.96.195
--webserver_port=0
--tserver_master_addrs=127.18.96.254:40433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.96.195
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:12.847918 7295 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:12.848901 7295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:12.854717 7301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:12.854719 7300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.854825 7295 server_base.cc:1061] running on GCE node
W20260501 02:09:12.854700 7303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:12.855167 7295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:12.855615 7295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:12.856781 7295 hybrid_clock.cc:648] HybridClock initialized: now 1777601352856762 us; error 28 us; skew 500 ppm
I20260501 02:09:12.858438 7295 webserver.cc:492] Webserver started at http://127.18.96.195:38137/ using document root <none> and password file <none>
I20260501 02:09:12.858943 7295 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:12.858992 7295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:12.859215 7295 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:12.860697 7295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data/instance:
uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "63d04d13ee566cbe05babc407d6c876e"
server_key_iv: "f8a91464e335674add96916256fcd327"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.861141 7295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance:
uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "63d04d13ee566cbe05babc407d6c876e"
server_key_iv: "f8a91464e335674add96916256fcd327"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.864071 7295 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:09:12.866027 7309 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.866827 7295 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:12.866950 7295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal
uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b"
format_stamp: "Formatted at 2026-05-01 02:09:12 on dist-test-slave-2j90"
server_key: "63d04d13ee566cbe05babc407d6c876e"
server_key_iv: "f8a91464e335674add96916256fcd327"
server_key_version: "encryptionkey@0"
I20260501 02:09:12.867040 7295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:12.885143 7295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:12.885730 7295 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:12.885907 7295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:12.886478 7295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260501 02:09:12.887349 7295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260501 02:09:12.887419 7295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.887490 7295 ts_tablet_manager.cc:616] Registered 0 tablets
I20260501 02:09:12.887534 7295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:12.896420 7295 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.195:37837
I20260501 02:09:12.896651 7422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.195:37837 every 8 connection(s)
I20260501 02:09:12.897567 7295 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/data/info.pb
I20260501 02:09:12.904266 7012 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.899360 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:47925 (local address 127.18.96.254:40433)
0501 02:09:12.899495 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.899498 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.900247 (+ 749us) server_negotiation.cc:408] Connection header received
0501 02:09:12.900996 (+ 749us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.900998 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.901045 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.901119 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:12.901870 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.902519 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.903092 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.903224 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.903871 (+ 647us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:12.903887 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:12.903889 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:12.903898 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:12.903913 (+ 15us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:12.903920 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:12.903979 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.904110 (+ 131us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.904151 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260501 02:09:12.904803 7425 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.899610 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:40433 (local address 127.18.96.195:47925)
0501 02:09:12.900112 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.900139 (+ 27us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.900836 (+ 697us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.901252 (+ 416us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.901260 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.901312 (+ 52us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:12.901753 (+ 441us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.901764 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.902625 (+ 861us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.902628 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.902998 (+ 370us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.903003 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.903143 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.903702 (+ 559us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0501 02:09:12.903703 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:12.903725 (+ 22us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0501 02:09:12.903765 (+ 40us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:12.903995 (+ 230us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:12.904000 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:12.904004 (+ 4us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.904099 (+ 95us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.904212 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260501 02:09:12.905755 7423 heartbeater.cc:344] Connected to a master server at 127.18.96.254:40433
I20260501 02:09:12.906010 7423 heartbeater.cc:461] Registering TS with master...
I20260501 02:09:12.906466 7423 heartbeater.cc:507] Master 127.18.96.254:40433 requested a full tablet report, sending...
I20260501 02:09:12.906579 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7295
I20260501 02:09:12.906710 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal/instance
I20260501 02:09:12.906948 18819 external_mini_cluster.cc:1468] Setting key 49fa6739c47c46942f90966a5746ad44
I20260501 02:09:12.907611 6973 ts_manager.cc:194] Registered new tserver with Master: 95b2239e3d9b4d1cab1f3cd8f62d6e4b (127.18.96.195:37837)
I20260501 02:09:12.908321 6973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.18.96.195:47925
I20260501 02:09:12.909003 18819 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260501 02:09:12.913496 7012 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.910777 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.18.96.254:40433)
0501 02:09:12.910918 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.910921 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.911096 (+ 175us) server_negotiation.cc:408] Connection header received
0501 02:09:12.911181 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.911183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.911217 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.911280 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:12.911626 (+ 346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.912039 (+ 413us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.912628 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.912744 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.913182 (+ 438us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:12.913194 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:12.913196 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:12.913205 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:12.913223 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:12.913229 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:12.913286 (+ 57us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.913373 (+ 87us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.913408 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:12.915851 6972 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260501 02:09:12.918170 6972 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260501 02:09:12.930094 7436 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.925928 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:38245 (local address 127.0.0.1:41594)
0501 02:09:12.926623 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.926636 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.926735 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.927275 (+ 540us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.927278 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.927296 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.927603 (+ 307us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.927613 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.928680 (+ 1067us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.928684 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.929552 (+ 868us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.929561 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.929669 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.929716 (+ 47us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.929796 (+ 80us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.929868 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":134,"threads_started":1}
I20260501 02:09:12.930104 7434 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.925379 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.195:37837 (local address 127.0.0.1:55002)
0501 02:09:12.925673 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.925708 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.925824 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.926628 (+ 804us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.926633 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.926657 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.926950 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.926958 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.928894 (+ 1936us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.928897 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.929716 (+ 819us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.929722 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.929813 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.929826 (+ 13us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.929871 (+ 45us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.929908 (+ 37us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260501 02:09:12.930661 7438 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.926293 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42771 (local address 127.0.0.1:54090)
0501 02:09:12.926910 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:12.926925 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:12.926991 (+ 66us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:12.927639 (+ 648us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:12.927645 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:12.927669 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:12.927905 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.927912 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.929231 (+ 1319us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.929235 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:12.930317 (+ 1082us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:12.930326 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.930440 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.930452 (+ 12us) client_negotiation.cc:770] Sending connection context
0501 02:09:12.930501 (+ 49us) client_negotiation.cc:241] Negotiation successful
0501 02:09:12.930546 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":92,"threads_started":1}
I20260501 02:09:12.931123 7435 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.925455 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.18.96.195:37837)
0501 02:09:12.926165 (+ 710us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.926173 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.926196 (+ 23us) server_negotiation.cc:408] Connection header received
0501 02:09:12.926270 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.926275 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.926442 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.927087 (+ 645us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.927677 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.928789 (+ 1112us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.929822 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.930640 (+ 818us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.930920 (+ 280us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.930967 (+ 47us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.931026 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":135,"threads_started":1}
I20260501 02:09:12.931351 7439 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.926451 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.18.96.194:42771)
0501 02:09:12.927195 (+ 744us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.927200 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.927221 (+ 21us) server_negotiation.cc:408] Connection header received
0501 02:09:12.927304 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.927311 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.927497 (+ 186us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.927642 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.928025 (+ 383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.929085 (+ 1060us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.930458 (+ 1373us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.930950 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.931067 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.931129 (+ 62us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.931205 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":123,"threads_started":1}
I20260501 02:09:12.932009 7437 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:12.926164 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.18.96.193:38245)
0501 02:09:12.926875 (+ 711us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:12.926881 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:12.926901 (+ 20us) server_negotiation.cc:408] Connection header received
0501 02:09:12.926983 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:12.926988 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:12.927137 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:12.927287 (+ 150us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:12.927757 (+ 470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.928565 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:12.930474 (+ 1909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:12.931070 (+ 596us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:12.931171 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:12.931792 (+ 621us) server_negotiation.cc:300] Negotiation successful
0501 02:09:12.931881 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":118,"threads_started":1}
I20260501 02:09:12.933636 7226 tablet_service.cc:1511] Processing CreateTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 (DEFAULT_TABLE table=test-table [id=d4485048b3c54b88b86da71c288c488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:12.933856 7357 tablet_service.cc:1511] Processing CreateTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 (DEFAULT_TABLE table=test-table [id=d4485048b3c54b88b86da71c288c488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:12.934548 7226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8e6f8297e3418a8082f5a0b6287615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:12.934425 7095 tablet_service.cc:1511] Processing CreateTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 (DEFAULT_TABLE table=test-table [id=d4485048b3c54b88b86da71c288c488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260501 02:09:12.934818 7357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8e6f8297e3418a8082f5a0b6287615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:12.935343 7095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae8e6f8297e3418a8082f5a0b6287615. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:12.939553 7440 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Bootstrap starting.
I20260501 02:09:12.940654 7441 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Bootstrap starting.
I20260501 02:09:12.941358 7442 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: Bootstrap starting.
I20260501 02:09:12.941772 7440 tablet_bootstrap.cc:654] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:12.942597 7441 tablet_bootstrap.cc:654] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:12.942621 7440 log.cc:826] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:12.943277 7442 tablet_bootstrap.cc:654] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:12.943284 7441 log.cc:826] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:12.943990 7442 log.cc:826] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:12.944478 7440 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: No bootstrap required, opened a new log
I20260501 02:09:12.944675 7440 ts_tablet_manager.cc:1403] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260501 02:09:12.945032 7441 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: No bootstrap required, opened a new log
I20260501 02:09:12.945200 7441 ts_tablet_manager.cc:1403] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260501 02:09:12.945570 7442 tablet_bootstrap.cc:492] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: No bootstrap required, opened a new log
I20260501 02:09:12.945752 7442 ts_tablet_manager.cc:1403] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260501 02:09:12.948223 7440 raft_consensus.cc:359] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.948309 7442 raft_consensus.cc:359] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.948434 7440 raft_consensus.cc:385] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:12.948475 7442 raft_consensus.cc:385] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:12.948503 7440 raft_consensus.cc:740] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5c1a0f71024a7083e575a09b543e04, State: Initialized, Role: FOLLOWER
I20260501 02:09:12.948518 7442 raft_consensus.cc:740] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf8faebff79470aa3c26b0f539267fc, State: Initialized, Role: FOLLOWER
I20260501 02:09:12.948459 7441 raft_consensus.cc:359] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.948638 7441 raft_consensus.cc:385] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:12.948698 7441 raft_consensus.cc:740] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95b2239e3d9b4d1cab1f3cd8f62d6e4b, State: Initialized, Role: FOLLOWER
I20260501 02:09:12.948966 7442 consensus_queue.cc:260] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.948962 7440 consensus_queue.cc:260] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.949000 7441 consensus_queue.cc:260] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:12.949568 7161 heartbeater.cc:499] Master 127.18.96.254:40433 was elected leader, sending a full tablet report...
I20260501 02:09:12.949746 7441 ts_tablet_manager.cc:1434] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260501 02:09:12.949759 7442 ts_tablet_manager.cc:1434] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260501 02:09:12.950088 7423 heartbeater.cc:499] Master 127.18.96.254:40433 was elected leader, sending a full tablet report...
I20260501 02:09:12.950850 7440 ts_tablet_manager.cc:1434] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260501 02:09:12.951118 7292 heartbeater.cc:499] Master 127.18.96.254:40433 was elected leader, sending a full tablet report...
W20260501 02:09:12.982146 7293 tablet.cc:2404] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:09:13.018675 7447 raft_consensus.cc:493] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260501 02:09:13.018877 7447 raft_consensus.cc:515] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:13.019902 7447 leader_election.cc:290] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adf8faebff79470aa3c26b0f539267fc (127.18.96.193:38245), 8c5c1a0f71024a7083e575a09b543e04 (127.18.96.194:42771)
I20260501 02:09:13.023803 7425 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.020109 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:38245 (local address 127.18.96.195:36745)
0501 02:09:13.020243 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:13.020260 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:13.020363 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:13.020694 (+ 331us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:13.020699 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:13.020719 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:13.020975 (+ 256us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:13.020981 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.022482 (+ 1501us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:13.022487 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:13.023401 (+ 914us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:13.023405 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.023520 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:13.023547 (+ 27us) client_negotiation.cc:770] Sending connection context
0501 02:09:13.023598 (+ 51us) client_negotiation.cc:241] Negotiation successful
0501 02:09:13.023648 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260501 02:09:13.023813 7449 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.020372 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.194:42771 (local address 127.18.96.195:60091)
0501 02:09:13.020785 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:13.020802 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:13.020886 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:13.021143 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:13.021145 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:13.021161 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0501 02:09:13.021360 (+ 199us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:13.021364 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.022461 (+ 1097us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:13.022467 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:13.023397 (+ 930us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:13.023405 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.023520 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:13.023547 (+ 27us) client_negotiation.cc:770] Sending connection context
0501 02:09:13.023606 (+ 59us) client_negotiation.cc:241] Negotiation successful
0501 02:09:13.023647 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":178,"threads_started":1}
I20260501 02:09:13.024281 7437 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.020227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:36745 (local address 127.18.96.193:38245)
0501 02:09:13.020371 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:13.020375 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:13.020391 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:09:13.020455 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:13.020460 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:13.020524 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:13.020646 (+ 122us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:13.021117 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.022337 (+ 1220us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:13.023521 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.024017 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:13.024047 (+ 30us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:13.024095 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:09:13.024153 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:09:13.024281 7439 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.020460 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.18.96.195:60091 (local address 127.18.96.194:42771)
0501 02:09:13.020622 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:13.020628 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:13.020884 (+ 256us) server_negotiation.cc:408] Connection header received
0501 02:09:13.020961 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:13.020963 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:13.021009 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:13.021121 (+ 112us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0501 02:09:13.021484 (+ 363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.022337 (+ 853us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:13.023521 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.024018 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:13.024047 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:13.024095 (+ 48us) server_negotiation.cc:300] Negotiation successful
0501 02:09:13.024153 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:09:13.024899 7115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae8e6f8297e3418a8082f5a0b6287615" candidate_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf8faebff79470aa3c26b0f539267fc" is_pre_election: true
I20260501 02:09:13.025058 7246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae8e6f8297e3418a8082f5a0b6287615" candidate_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5c1a0f71024a7083e575a09b543e04" is_pre_election: true
I20260501 02:09:13.025192 7115 raft_consensus.cc:2468] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b2239e3d9b4d1cab1f3cd8f62d6e4b in term 0.
I20260501 02:09:13.025297 7246 raft_consensus.cc:2468] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95b2239e3d9b4d1cab1f3cd8f62d6e4b in term 0.
I20260501 02:09:13.025694 7313 leader_election.cc:304] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b2239e3d9b4d1cab1f3cd8f62d6e4b, adf8faebff79470aa3c26b0f539267fc; no voters:
I20260501 02:09:13.025918 7447 raft_consensus.cc:2804] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260501 02:09:13.026036 7447 raft_consensus.cc:493] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260501 02:09:13.026093 7447 raft_consensus.cc:3060] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:13.027027 7447 raft_consensus.cc:515] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:13.027365 7447 leader_election.cc:290] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [CANDIDATE]: Term 1 election: Requested vote from peers adf8faebff79470aa3c26b0f539267fc (127.18.96.193:38245), 8c5c1a0f71024a7083e575a09b543e04 (127.18.96.194:42771)
I20260501 02:09:13.027751 7115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae8e6f8297e3418a8082f5a0b6287615" candidate_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf8faebff79470aa3c26b0f539267fc"
I20260501 02:09:13.027804 7246 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae8e6f8297e3418a8082f5a0b6287615" candidate_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5c1a0f71024a7083e575a09b543e04"
I20260501 02:09:13.027860 7115 raft_consensus.cc:3060] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:13.027944 7246 raft_consensus.cc:3060] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:13.028820 7115 raft_consensus.cc:2468] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b2239e3d9b4d1cab1f3cd8f62d6e4b in term 1.
I20260501 02:09:13.028913 7246 raft_consensus.cc:2468] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95b2239e3d9b4d1cab1f3cd8f62d6e4b in term 1.
I20260501 02:09:13.029093 7313 leader_election.cc:304] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95b2239e3d9b4d1cab1f3cd8f62d6e4b, adf8faebff79470aa3c26b0f539267fc; no voters:
I20260501 02:09:13.029258 7447 raft_consensus.cc:2804] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:13.029471 7447 raft_consensus.cc:697] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 1 LEADER]: Becoming Leader. State: Replica: 95b2239e3d9b4d1cab1f3cd8f62d6e4b, State: Running, Role: LEADER
I20260501 02:09:13.029788 7447 consensus_queue.cc:237] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } }
I20260501 02:09:13.032774 6971 catalog_manager.cc:5671] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 95b2239e3d9b4d1cab1f3cd8f62d6e4b (127.18.96.195). New cstate: current_term: 1 leader_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95b2239e3d9b4d1cab1f3cd8f62d6e4b" member_type: VOTER last_known_addr { host: "127.18.96.195" port: 37837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 } health_report { overall_health: UNKNOWN } } }
I20260501 02:09:13.055688 7435 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.052773 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.18.96.195:37837)
0501 02:09:13.052901 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:13.052904 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:13.052917 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:09:13.052981 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:13.052984 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:13.053036 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:13.053112 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:13.053446 (+ 334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.054029 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:13.054703 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:13.054908 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:13.054979 (+ 71us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:13.055291 (+ 312us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:13.055419 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:13.055514 (+ 95us) server_negotiation.cc:300] Negotiation successful
0501 02:09:13.055558 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:09:13.060838 7115 raft_consensus.cc:1275] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 1 FOLLOWER]: Refusing update from remote peer 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:13.060838 7246 raft_consensus.cc:1275] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 1 FOLLOWER]: Refusing update from remote peer 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:13.061496 7447 consensus_queue.cc:1048] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5c1a0f71024a7083e575a09b543e04" member_type: VOTER last_known_addr { host: "127.18.96.194" port: 42771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:13.061673 7450 consensus_queue.cc:1048] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf8faebff79470aa3c26b0f539267fc" member_type: VOTER last_known_addr { host: "127.18.96.193" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:13.068486 7454 mvcc.cc:204] Tried to move back new op lower bound from 7281055142131015680 to 7281055142013022208. Current Snapshot: MvccSnapshot[applied={T|T < 7281055142131015680}]
I20260501 02:09:13.069774 7455 mvcc.cc:204] Tried to move back new op lower bound from 7281055142131015680 to 7281055142013022208. Current Snapshot: MvccSnapshot[applied={T|T < 7281055142131015680}]
I20260501 02:09:13.076244 6970 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:13.076375 6970 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260501 02:09:13.078584 6970 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 283d6cff9ab94729902347b615fd20c9: Sending DeleteTablet for 3 replicas of tablet ae8e6f8297e3418a8082f5a0b6287615
I20260501 02:09:13.079385 7226 tablet_service.cc:1558] Processing DeleteTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:13 UTC) from {username='slave'} at 127.0.0.1:54090
I20260501 02:09:13.079373 7095 tablet_service.cc:1558] Processing DeleteTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:13 UTC) from {username='slave'} at 127.0.0.1:41594
I20260501 02:09:13.079802 7462 tablet_replica.cc:333] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc: stopping tablet replica
I20260501 02:09:13.079805 7357 tablet_service.cc:1558] Processing DeleteTablet for tablet ae8e6f8297e3418a8082f5a0b6287615 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-05-01 02:09:13 UTC) from {username='slave'} at 127.0.0.1:55002
I20260501 02:09:13.079936 7463 tablet_replica.cc:333] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: stopping tablet replica
I20260501 02:09:13.080165 7462 raft_consensus.cc:2243] T ae8e6f8297e3418a8082f5a0b6287615 P adf8faebff79470aa3c26b0f539267fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:13.080165 7463 raft_consensus.cc:2243] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260501 02:09:13.080469 7463 raft_consensus.cc:2272] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:13.080644 7464 tablet_replica.cc:333] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: stopping tablet replica
I20260501 02:09:13.080746 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7033
I20260501 02:09:13.080844 7464 raft_consensus.cc:2243] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 1 LEADER]: Raft consensus shutting down.
I20260501 02:09:13.081151 7464 raft_consensus.cc:2272] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260501 02:09:13.082670 7464 ts_tablet_manager.cc:1916] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:13.082670 7463 ts_tablet_manager.cc:1916] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260501 02:09:13.085111 7463 ts_tablet_manager.cc:1929] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:13.085119 7464 ts_tablet_manager.cc:1929] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260501 02:09:13.085201 7463 log.cc:1199] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-1/wal/wals/ae8e6f8297e3418a8082f5a0b6287615
I20260501 02:09:13.085201 7464 log.cc:1199] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Deleting WAL directory at /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1777600936179445-18819-0/minicluster-data/ts-2/wal/wals/ae8e6f8297e3418a8082f5a0b6287615
I20260501 02:09:13.085536 7463 ts_tablet_manager.cc:1950] T ae8e6f8297e3418a8082f5a0b6287615 P 8c5c1a0f71024a7083e575a09b543e04: Deleting consensus metadata
I20260501 02:09:13.085520 7464 ts_tablet_manager.cc:1950] T ae8e6f8297e3418a8082f5a0b6287615 P 95b2239e3d9b4d1cab1f3cd8f62d6e4b: Deleting consensus metadata
I20260501 02:09:13.086614 6960 catalog_manager.cc:5002] TS 95b2239e3d9b4d1cab1f3cd8f62d6e4b (127.18.96.195:37837): tablet ae8e6f8297e3418a8082f5a0b6287615 (table test-table [id=d4485048b3c54b88b86da71c288c488c]) successfully deleted
I20260501 02:09:13.086838 6960 catalog_manager.cc:5002] TS 8c5c1a0f71024a7083e575a09b543e04 (127.18.96.194:42771): tablet ae8e6f8297e3418a8082f5a0b6287615 (table test-table [id=d4485048b3c54b88b86da71c288c488c]) successfully deleted
W20260501 02:09:13.091246 6960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.96.193:38245 (error 108)
I20260501 02:09:13.091507 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7164
I20260501 02:09:13.093640 7438 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:13.093299 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.193:38245 (local address 127.0.0.1:41606)
0501 02:09:13.093449 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:13.093521 (+ 72us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.193:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260501 02:09:13.093803 6960 catalog_manager.cc:4729] TS adf8faebff79470aa3c26b0f539267fc (127.18.96.193:38245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae8e6f8297e3418a8082f5a0b6287615: Network error: Client connection negotiation failed: client connection to 127.18.96.193:38245: connect: Connection refused (error 111)
I20260501 02:09:13.097502 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7295
I20260501 02:09:13.103022 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 6942
2026-05-01T02:09:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (944 ms)
[----------] 6 tests from EncryptionPolicyTest (13102 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:13 dist-test-slave-2j90 krb5kdc[7468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:13 dist-test-slave-2j90 krb5kdc[7468](info): set up 2 sockets
May 01 02:09:13 dist-test-slave-2j90 krb5kdc[7468](info): commencing operation
krb5kdc: starting...
W20260501 02:09:15.074750 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:15.212888 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:15.312418 7484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:15.312618 7484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:15.312666 7484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:15.315904 7484 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:15.315958 7484 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:15.315980 7484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:15.315999 7484 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:15.316045 7484 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:15.320043 7484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:15.320940 7484 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:15.321717 7484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:15.327020 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:15.327040 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:15.327082 7484 server_base.cc:1061] running on GCE node
W20260501 02:09:15.326997 7492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:15.327776 7484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:15.328608 7484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:15.329777 7484 hybrid_clock.cc:648] HybridClock initialized: now 1777601355329748 us; error 40 us; skew 500 ppm
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:15.332459 7484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:15.333480 7484 webserver.cc:492] Webserver started at http://127.0.0.1:41333/ using document root <none> and password file <none>
I20260501 02:09:15.334079 7484 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:15.334151 7484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:15.334350 7484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:15.335903 7484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "e0b554f904db41cfacbd481ad5c712df"
format_stamp: "Formatted at 2026-05-01 02:09:15 on dist-test-slave-2j90"
server_key: "1047ea6370f5c7f1b300c633dc4ca769"
server_key_iv: "b02551c9083901ef00a14e5a37e3e802"
server_key_version: "encryptionkey@0"
I20260501 02:09:15.336376 7484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "e0b554f904db41cfacbd481ad5c712df"
format_stamp: "Formatted at 2026-05-01 02:09:15 on dist-test-slave-2j90"
server_key: "1047ea6370f5c7f1b300c633dc4ca769"
server_key_iv: "b02551c9083901ef00a14e5a37e3e802"
server_key_version: "encryptionkey@0"
I20260501 02:09:15.339525 7484 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:15.341472 7499 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:15.342540 7484 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:15.342681 7484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "e0b554f904db41cfacbd481ad5c712df"
format_stamp: "Formatted at 2026-05-01 02:09:15 on dist-test-slave-2j90"
server_key: "1047ea6370f5c7f1b300c633dc4ca769"
server_key_iv: "b02551c9083901ef00a14e5a37e3e802"
server_key_version: "encryptionkey@0"
I20260501 02:09:15.342796 7484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:15.361454 7484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:15.363759 7484 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:15.363936 7484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:15.370693 7484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42239
I20260501 02:09:15.370690 7551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42239 every 8 connection(s)
I20260501 02:09:15.371641 7484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:15.374135 7552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:15.378094 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7484
I20260501 02:09:15.378229 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:15.378509 18819 external_mini_cluster.cc:1468] Setting key 3a6dc0495adfeddb992aec19f6668d43
I20260501 02:09:15.379379 7552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: Bootstrap starting.
I20260501 02:09:15.381551 7552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:15.382380 7552 log.cc:826] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:15.384200 7552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: No bootstrap required, opened a new log
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260501 02:09:15.387311 7552 raft_consensus.cc:359] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260501 02:09:15.387480 7552 raft_consensus.cc:385] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:15.387562 7552 raft_consensus.cc:740] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b554f904db41cfacbd481ad5c712df, State: Initialized, Role: FOLLOWER
I20260501 02:09:15.387969 7552 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260501 02:09:15.388104 7552 raft_consensus.cc:399] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:15.388183 7552 raft_consensus.cc:493] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:15.388294 7552 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:15.389060 7552 raft_consensus.cc:515] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260501 02:09:15.389421 7552 leader_election.cc:304] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b554f904db41cfacbd481ad5c712df; no voters:
I20260501 02:09:15.389767 7552 leader_election.cc:290] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:15.389874 7557 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:15.390103 7557 raft_consensus.cc:697] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [term 1 LEADER]: Becoming Leader. State: Replica: e0b554f904db41cfacbd481ad5c712df, State: Running, Role: LEADER
I20260501 02:09:15.390394 7557 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } }
I20260501 02:09:15.390890 7552 sys_catalog.cc:565] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:15.391589 7557 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0b554f904db41cfacbd481ad5c712df. Latest consensus state: current_term: 1 leader_uuid: "e0b554f904db41cfacbd481ad5c712df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260501 02:09:15.391692 7557 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:15.392134 7558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0b554f904db41cfacbd481ad5c712df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b554f904db41cfacbd481ad5c712df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42239 } } }
I20260501 02:09:15.392386 7558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:15.392688 7555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:15.379700 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46267 (local address 127.0.0.1:42239)
0501 02:09:15.380080 (+ 380us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:15.380089 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:15.380119 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:09:15.380637 (+ 518us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:15.380651 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:15.380896 (+ 245us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:15.381129 (+ 233us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:15.382247 (+ 1118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:15.383278 (+ 1031us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:15.384173 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:15.384526 (+ 353us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:15.386985 (+ 2459us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:15.387002 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:15.387013 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:15.387039 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:15.389156 (+ 2117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:15.389846 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:15.389852 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:15.389859 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:15.389984 (+ 125us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:15.390374 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:15.390377 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:15.390379 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:15.390791 (+ 412us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:15.391499 (+ 708us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:15.391690 (+ 191us) server_negotiation.cc:300] Negotiation successful
0501 02:09:15.391900 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":129,"threads_started":1}
I20260501 02:09:15.395478 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260501 02:09:15.395890 7572 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:15.395978 7572 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:15.396044 7563 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:15.396934 7563 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:15.402978 7563 catalog_manager.cc:1357] Generated new cluster ID: 7a2dd4dd26fc421cb1bc1d3dac1908d8
I20260501 02:09:15.403064 7563 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:15.405699 7555 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:15.397586 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40673 (local address 127.0.0.1:42239)
0501 02:09:15.398117 (+ 531us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:15.398122 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:15.398137 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:09:15.398174 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:15.398177 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:15.398221 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:15.398312 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:15.399205 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:15.399674 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:15.400264 (+ 590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:15.400469 (+ 205us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:15.401814 (+ 1345us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:15.401829 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:15.401833 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:15.401857 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:15.403615 (+ 1758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:15.404277 (+ 662us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:15.404281 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:15.404283 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:15.404332 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:15.404708 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:15.404714 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:15.404717 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:15.404915 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:15.405042 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:15.405442 (+ 400us) server_negotiation.cc:300] Negotiation successful
0501 02:09:15.405534 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451}
I20260501 02:09:15.429149 7563 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:15.429878 7563 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:15.435642 7563 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e0b554f904db41cfacbd481ad5c712df: Generated new TSK 0
I20260501 02:09:15.436117 7563 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:15.441996 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7484
W20260501 02:09:15.447794 7466 connection.cc:570] client connection to 127.0.0.1:42239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42239): Connection reset by peer (error 104)
2026-05-01T02:09:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7583](info): set up 2 sockets
May 01 02:09:15 dist-test-slave-2j90 krb5kdc[7583](info): commencing operation
krb5kdc: starting...
W20260501 02:09:17.430526 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:17 dist-test-slave-2j90 krb5kdc[7583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:17.562922 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:17.662150 7599 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:17.662390 7599 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:17.662478 7599 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:17.666183 7599 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:17.666278 7599 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:17.666356 7599 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:17.666534 7599 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:17.666626 7599 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:17.670518 7599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:17.671360 7599 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:17.672156 7599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:17.677222 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:17.677227 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:17.677227 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:17.677505 7599 server_base.cc:1061] running on GCE node
I20260501 02:09:17.677919 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:17.678745 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:17.679917 7599 hybrid_clock.cc:648] HybridClock initialized: now 1777601357679889 us; error 50 us; skew 500 ppm
May 01 02:09:17 dist-test-slave-2j90 krb5kdc[7583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:17.682461 7599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:17.683342 7599 webserver.cc:492] Webserver started at http://127.0.0.1:33587/ using document root <none> and password file <none>
I20260501 02:09:17.683845 7599 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:17.683917 7599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:17.684106 7599 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:17.685637 7599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d"
format_stamp: "Formatted at 2026-05-01 02:09:17 on dist-test-slave-2j90"
server_key: "2825dae23b643da9d1321063e4893447"
server_key_iv: "d8cb6b0a40bfcbbc1c610698c206e5c1"
server_key_version: "encryptionkey@0"
I20260501 02:09:17.686146 7599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d"
format_stamp: "Formatted at 2026-05-01 02:09:17 on dist-test-slave-2j90"
server_key: "2825dae23b643da9d1321063e4893447"
server_key_iv: "d8cb6b0a40bfcbbc1c610698c206e5c1"
server_key_version: "encryptionkey@0"
I20260501 02:09:17.689167 7599 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260501 02:09:17.691099 7614 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:17.691888 7599 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:17.692010 7599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d"
format_stamp: "Formatted at 2026-05-01 02:09:17 on dist-test-slave-2j90"
server_key: "2825dae23b643da9d1321063e4893447"
server_key_iv: "d8cb6b0a40bfcbbc1c610698c206e5c1"
server_key_version: "encryptionkey@0"
I20260501 02:09:17.692097 7599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:17.707389 7599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:17.709451 7599 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:17.709630 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:17.716008 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44643
I20260501 02:09:17.716022 7666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:44643 every 8 connection(s)
I20260501 02:09:17.716886 7599 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:17.717401 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7599
I20260501 02:09:17.717540 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:17.717804 18819 external_mini_cluster.cc:1468] Setting key 020ff0c8114e1783fb183a49cea31e6d
I20260501 02:09:17.719928 7667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:17.725528 7667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d: Bootstrap starting.
I20260501 02:09:17.725639 7670 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:17.718848 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48961 (local address 127.0.0.1:44643)
0501 02:09:17.719189 (+ 341us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:17.719199 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:17.719250 (+ 51us) server_negotiation.cc:408] Connection header received
0501 02:09:17.720009 (+ 759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:17.720031 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:17.720341 (+ 310us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:17.720651 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:17.721057 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:17.722716 (+ 1659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:17.723401 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:17.723650 (+ 249us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:17.724063 (+ 413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:17.724118 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:17.724130 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:17.724582 (+ 452us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:17.724620 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:17.724642 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:17.724776 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:17.724930 (+ 154us) server_negotiation.cc:300] Negotiation successful
0501 02:09:17.725013 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260501 02:09:17.727711 7667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:17.728111 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:17.728431 7667 log.cc:826] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:17.730211 7667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d: No bootstrap required, opened a new log
I20260501 02:09:17.733160 7667 raft_consensus.cc:359] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260501 02:09:17.733369 7667 raft_consensus.cc:385] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:17.733443 7667 raft_consensus.cc:740] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30bf3ecfdfbd4ecab0ecf9cb877c5e1d, State: Initialized, Role: FOLLOWER
I20260501 02:09:17.733508 7670 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:17.730000 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45105 (local address 127.0.0.1:44643)
0501 02:09:17.730105 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:17.730108 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:17.730198 (+ 90us) server_negotiation.cc:408] Connection header received
0501 02:09:17.730316 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:17.730320 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:17.730369 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:17.730434 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:17.730927 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:17.731354 (+ 427us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:17.732139 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:17.732277 (+ 138us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:17.732808 (+ 531us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:17.732821 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:17.732823 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:17.732833 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:17.732855 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:17.732864 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:17.732930 (+ 66us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:17.733079 (+ 149us) server_negotiation.cc:300] Negotiation successful
0501 02:09:17.733100 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260501 02:09:17.733816 7667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260501 02:09:17.733947 7667 raft_consensus.cc:399] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:17.734031 7667 raft_consensus.cc:493] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:17.734130 7667 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:17.735246 7667 raft_consensus.cc:515] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260501 02:09:17.735554 7667 leader_election.cc:304] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30bf3ecfdfbd4ecab0ecf9cb877c5e1d; no voters:
I20260501 02:09:17.735823 7667 leader_election.cc:290] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:17.735949 7677 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:17.736248 7677 raft_consensus.cc:697] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [term 1 LEADER]: Becoming Leader. State: Replica: 30bf3ecfdfbd4ecab0ecf9cb877c5e1d, State: Running, Role: LEADER
I20260501 02:09:17.736588 7677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } }
I20260501 02:09:17.736881 7667 sys_catalog.cc:565] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:17.738195 7678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260501 02:09:17.738371 7678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:17.738405 7679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30bf3ecfdfbd4ecab0ecf9cb877c5e1d. Latest consensus state: current_term: 1 leader_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30bf3ecfdfbd4ecab0ecf9cb877c5e1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44643 } } }
I20260501 02:09:17.738500 7679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:17.738778 7686 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:17.741443 7686 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:17.746747 7686 catalog_manager.cc:1357] Generated new cluster ID: 8ef875f587294fbf91aadca8fc19c207
I20260501 02:09:17.746835 7686 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:17.753572 7686 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:17.754573 7686 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:17.762569 7686 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 30bf3ecfdfbd4ecab0ecf9cb877c5e1d: Generated new TSK 0
I20260501 02:09:17.763090 7686 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:17.766765 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7599
W20260501 02:09:17.772321 7581 connection.cc:570] client connection to 127.0.0.1:44643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44643): Connection reset by peer (error 104)
2026-05-01T02:09:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:17 dist-test-slave-2j90 krb5kdc[7698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:17 dist-test-slave-2j90 krb5kdc[7698](info): set up 2 sockets
May 01 02:09:17 dist-test-slave-2j90 krb5kdc[7698](info): commencing operation
krb5kdc: starting...
W20260501 02:09:19.766870 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:19 dist-test-slave-2j90 krb5kdc[7698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:19.902462 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:19.999215 7714 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:19.999450 7714 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:19.999529 7714 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:20.002753 7714 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:20.002840 7714 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:20.002892 7714 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:20.002969 7714 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:20.003010 7714 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:20.007148 7714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:20.008108 7714 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:20.008863 7714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:20.014215 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:20.014225 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:20.014225 7722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:20.014277 7714 server_base.cc:1061] running on GCE node
I20260501 02:09:20.014915 7714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:20.015667 7714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:20.016813 7714 hybrid_clock.cc:648] HybridClock initialized: now 1777601360016791 us; error 34 us; skew 500 ppm
May 01 02:09:20 dist-test-slave-2j90 krb5kdc[7698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:20.019438 7714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:20.020335 7714 webserver.cc:492] Webserver started at http://127.0.0.1:37205/ using document root <none> and password file <none>
I20260501 02:09:20.020787 7714 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:20.020831 7714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:20.020968 7714 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:20.022456 7714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "15a9985260b140c79bc861cee2c459d4"
format_stamp: "Formatted at 2026-05-01 02:09:20 on dist-test-slave-2j90"
server_key: "7fbf152aefab2f78cb113cb582a098b6"
server_key_iv: "58907c46c074e893f854eb9e49d8ce12"
server_key_version: "encryptionkey@0"
I20260501 02:09:20.022877 7714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "15a9985260b140c79bc861cee2c459d4"
format_stamp: "Formatted at 2026-05-01 02:09:20 on dist-test-slave-2j90"
server_key: "7fbf152aefab2f78cb113cb582a098b6"
server_key_iv: "58907c46c074e893f854eb9e49d8ce12"
server_key_version: "encryptionkey@0"
I20260501 02:09:20.025831 7714 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:20.027697 7729 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:20.028599 7714 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:20.028710 7714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "15a9985260b140c79bc861cee2c459d4"
format_stamp: "Formatted at 2026-05-01 02:09:20 on dist-test-slave-2j90"
server_key: "7fbf152aefab2f78cb113cb582a098b6"
server_key_iv: "58907c46c074e893f854eb9e49d8ce12"
server_key_version: "encryptionkey@0"
I20260501 02:09:20.028797 7714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:20.032821 7714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:20.038403 7714 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:20.038589 7714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:20.045289 7714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33755
I20260501 02:09:20.045307 7781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33755 every 8 connection(s)
I20260501 02:09:20.046367 7714 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:20.047155 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7714
I20260501 02:09:20.047312 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:20.047610 18819 external_mini_cluster.cc:1468] Setting key 55953f00c5810552e13b169fa88ab29c
I20260501 02:09:20.049561 7782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:20.052698 7785 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:20.048827 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:53495 (local address 127.0.0.1:33755)
0501 02:09:20.049298 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:20.049309 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:20.049342 (+ 33us) server_negotiation.cc:408] Connection header received
0501 02:09:20.050281 (+ 939us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:20.050299 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:20.050514 (+ 215us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:20.050765 (+ 251us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:20.051785 (+ 1020us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:20.051789 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:20.051797 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:20.051803 (+ 6us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:20.051823 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:20.051834 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:20.051946 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:20.052066 (+ 120us) server_negotiation.cc:300] Negotiation successful
0501 02:09:20.052184 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
I20260501 02:09:20.054777 7782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4: Bootstrap starting.
I20260501 02:09:20.055034 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:20.057288 7782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:20.058197 7782 log.cc:826] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:20.058773 7785 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:20.057146 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52083 (local address 127.0.0.1:33755)
0501 02:09:20.057305 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:20.057308 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:20.057411 (+ 103us) server_negotiation.cc:408] Connection header received
0501 02:09:20.057525 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:20.057528 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:20.057556 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:20.057626 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:20.058336 (+ 710us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:20.058342 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:20.058345 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:20.058356 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:20.058389 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:20.058400 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:20.058469 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:20.058629 (+ 160us) server_negotiation.cc:300] Negotiation successful
0501 02:09:20.058660 (+ 31us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260501 02:09:20.059890 7782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4: No bootstrap required, opened a new log
I20260501 02:09:20.062119 7782 raft_consensus.cc:359] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } }
I20260501 02:09:20.062278 7782 raft_consensus.cc:385] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:20.062310 7782 raft_consensus.cc:740] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a9985260b140c79bc861cee2c459d4, State: Initialized, Role: FOLLOWER
I20260501 02:09:20.062661 7782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } }
I20260501 02:09:20.062794 7782 raft_consensus.cc:399] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:20.062870 7782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:20.063014 7782 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:20.063793 7782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } }
I20260501 02:09:20.064090 7782 leader_election.cc:304] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15a9985260b140c79bc861cee2c459d4; no voters:
I20260501 02:09:20.064502 7782 leader_election.cc:290] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:20.064637 7792 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:20.064860 7792 raft_consensus.cc:697] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [term 1 LEADER]: Becoming Leader. State: Replica: 15a9985260b140c79bc861cee2c459d4, State: Running, Role: LEADER
I20260501 02:09:20.065162 7792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } }
I20260501 02:09:20.065518 7782 sys_catalog.cc:565] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:20.066861 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15a9985260b140c79bc861cee2c459d4. Latest consensus state: current_term: 1 leader_uuid: "15a9985260b140c79bc861cee2c459d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } } }
I20260501 02:09:20.066992 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:20.067234 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15a9985260b140c79bc861cee2c459d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15a9985260b140c79bc861cee2c459d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33755 } } }
I20260501 02:09:20.067302 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:20.068022 7801 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:20.070451 7801 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:20.075222 7801 catalog_manager.cc:1357] Generated new cluster ID: e2f6dd84c0ed458ea317d3eebc5d1d6c
I20260501 02:09:20.075290 7801 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:20.083056 7801 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:20.083312 7801 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:20.091534 7801 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 15a9985260b140c79bc861cee2c459d4: Generated new TSK 0
I20260501 02:09:20.092031 7801 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:20.102849 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7714
2026-05-01T02:09:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:20 dist-test-slave-2j90 krb5kdc[7813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:20 dist-test-slave-2j90 krb5kdc[7813](info): set up 2 sockets
May 01 02:09:20 dist-test-slave-2j90 krb5kdc[7813](info): commencing operation
krb5kdc: starting...
W20260501 02:09:22.074554 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:22.206894 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:22.304459 7829 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:22.304672 7829 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:22.304719 7829 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:22.307837 7829 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:22.307889 7829 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:22.307911 7829 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:22.307928 7829 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:22.307945 7829 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:22.311820 7829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:22.312624 7829 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:22.313319 7829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:22.318289 7834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:22.318432 7837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:22.318315 7835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:22.318749 7829 server_base.cc:1061] running on GCE node
I20260501 02:09:22.319192 7829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:22.319967 7829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:22.321139 7829 hybrid_clock.cc:648] HybridClock initialized: now 1777601362321115 us; error 42 us; skew 500 ppm
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:22.323752 7829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:22.324669 7829 webserver.cc:492] Webserver started at http://127.0.0.1:44707/ using document root <none> and password file <none>
I20260501 02:09:22.325176 7829 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:22.325244 7829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:22.325469 7829 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:22.327041 7829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "94fbffe52fac4a57af559b2230f2cf75"
format_stamp: "Formatted at 2026-05-01 02:09:22 on dist-test-slave-2j90"
server_key: "cfa313f4e0b89f483bc01a535f8d514c"
server_key_iv: "d4a660ec9f40840e6a4d05a45f41f385"
server_key_version: "encryptionkey@0"
I20260501 02:09:22.327512 7829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "94fbffe52fac4a57af559b2230f2cf75"
format_stamp: "Formatted at 2026-05-01 02:09:22 on dist-test-slave-2j90"
server_key: "cfa313f4e0b89f483bc01a535f8d514c"
server_key_iv: "d4a660ec9f40840e6a4d05a45f41f385"
server_key_version: "encryptionkey@0"
I20260501 02:09:22.330462 7829 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:22.332345 7844 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:22.333184 7829 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:22.333304 7829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "94fbffe52fac4a57af559b2230f2cf75"
format_stamp: "Formatted at 2026-05-01 02:09:22 on dist-test-slave-2j90"
server_key: "cfa313f4e0b89f483bc01a535f8d514c"
server_key_iv: "d4a660ec9f40840e6a4d05a45f41f385"
server_key_version: "encryptionkey@0"
I20260501 02:09:22.333400 7829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:22.357913 7829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:22.360265 7829 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:22.360455 7829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:22.367058 7829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34627
I20260501 02:09:22.367072 7896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34627 every 8 connection(s)
I20260501 02:09:22.367931 7829 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:22.370422 7897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:22.371793 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7829
I20260501 02:09:22.371933 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:22.372160 18819 external_mini_cluster.cc:1468] Setting key e58939deca92b56211ea307975a77b66
I20260501 02:09:22.376073 7897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75: Bootstrap starting.
I20260501 02:09:22.378183 7897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:22.378901 7897 log.cc:826] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75: Log is configured to *not* fsync() on all Append() calls
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260501 02:09:22.380939 7897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75: No bootstrap required, opened a new log
I20260501 02:09:22.383909 7897 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260501 02:09:22.384104 7897 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:22.384141 7900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:22.373200 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45037 (local address 127.0.0.1:34627)
0501 02:09:22.373595 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:22.373615 (+ 20us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:22.373652 (+ 37us) server_negotiation.cc:408] Connection header received
0501 02:09:22.374315 (+ 663us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:22.374348 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:22.374677 (+ 329us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:22.374962 (+ 285us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:22.375669 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:22.376739 (+ 1070us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:22.377337 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:22.377576 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:22.379923 (+ 2347us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:22.379940 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:22.379951 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:22.379977 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:22.381817 (+ 1840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:22.382245 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:22.382251 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:22.382256 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:22.382330 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:22.382537 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:22.382540 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:22.382542 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:22.382896 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:22.383078 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:22.383276 (+ 198us) server_negotiation.cc:300] Negotiation successful
0501 02:09:22.383486 (+ 210us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260501 02:09:22.384187 7897 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fbffe52fac4a57af559b2230f2cf75, State: Initialized, Role: FOLLOWER
I20260501 02:09:22.384603 7897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260501 02:09:22.384740 7897 raft_consensus.cc:399] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:22.384795 7897 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:22.384876 7897 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:22.386291 7897 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260501 02:09:22.386617 7897 leader_election.cc:304] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fbffe52fac4a57af559b2230f2cf75; no voters:
I20260501 02:09:22.386757 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:22.386929 7897 leader_election.cc:290] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:22.386992 7902 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:22.387159 7902 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [term 1 LEADER]: Becoming Leader. State: Replica: 94fbffe52fac4a57af559b2230f2cf75, State: Running, Role: LEADER
I20260501 02:09:22.387463 7902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260501 02:09:22.387976 7897 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:22.388563 7902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94fbffe52fac4a57af559b2230f2cf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } } }
I20260501 02:09:22.388777 7902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:22.388931 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fbffe52fac4a57af559b2230f2cf75. Latest consensus state: current_term: 1 leader_uuid: "94fbffe52fac4a57af559b2230f2cf75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fbffe52fac4a57af559b2230f2cf75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } } }
I20260501 02:09:22.389029 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:22.389194 7914 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:22.392138 7914 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:22.397024 7900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:22.389097 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35195 (local address 127.0.0.1:34627)
0501 02:09:22.389207 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:22.389212 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:22.389430 (+ 218us) server_negotiation.cc:408] Connection header received
0501 02:09:22.389565 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:22.389569 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:22.389611 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:22.389687 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:22.390715 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:22.391385 (+ 670us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:22.392189 (+ 804us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:22.392399 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:22.393850 (+ 1451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:22.393865 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:22.393867 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:22.393892 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:22.395486 (+ 1594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:22.395899 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:22.395902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:22.395904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:22.395953 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:22.396240 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:22.396243 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:22.396245 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:22.396386 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:22.396533 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:22.396737 (+ 204us) server_negotiation.cc:300] Negotiation successful
0501 02:09:22.396838 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260501 02:09:22.397375 7914 catalog_manager.cc:1357] Generated new cluster ID: cbab75a7b436469c8cb08b6e0f89e3a7
I20260501 02:09:22.397466 7914 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:22.429304 7914 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:22.430164 7914 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:22.448992 7914 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 94fbffe52fac4a57af559b2230f2cf75: Generated new TSK 0
I20260501 02:09:22.449636 7914 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:22.462080 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7829
2026-05-01T02:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7928](info): set up 2 sockets
May 01 02:09:22 dist-test-slave-2j90 krb5kdc[7928](info): commencing operation
krb5kdc: starting...
W20260501 02:09:24.434499 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:24 dist-test-slave-2j90 krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:24.564707 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:24.661332 7944 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:24.661585 7944 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:24.661665 7944 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:24.664860 7944 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:24.664939 7944 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:24.664979 7944 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:24.665033 7944 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:24.665071 7944 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:24.669081 7944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:24.669914 7944 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:24.670674 7944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:24.675678 7952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:24.675704 7949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:24.675678 7950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:24.675927 7944 server_base.cc:1061] running on GCE node
I20260501 02:09:24.676409 7944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:24.677217 7944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:24.678401 7944 hybrid_clock.cc:648] HybridClock initialized: now 1777601364678383 us; error 37 us; skew 500 ppm
May 01 02:09:24 dist-test-slave-2j90 krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:24.680936 7944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:24.681797 7944 webserver.cc:492] Webserver started at http://127.0.0.1:38927/ using document root <none> and password file <none>
I20260501 02:09:24.682351 7944 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:24.682399 7944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:24.682592 7944 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:24.684063 7944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "9cb8165d6e7b4a02872912df28ae1a9d"
format_stamp: "Formatted at 2026-05-01 02:09:24 on dist-test-slave-2j90"
server_key: "cdf1598885d08790a39fa730c90c53b6"
server_key_iv: "1e9d6ffb6728c472cef19407c2304ebc"
server_key_version: "encryptionkey@0"
I20260501 02:09:24.684486 7944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "9cb8165d6e7b4a02872912df28ae1a9d"
format_stamp: "Formatted at 2026-05-01 02:09:24 on dist-test-slave-2j90"
server_key: "cdf1598885d08790a39fa730c90c53b6"
server_key_iv: "1e9d6ffb6728c472cef19407c2304ebc"
server_key_version: "encryptionkey@0"
I20260501 02:09:24.687548 7944 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:24.689491 7959 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:24.690412 7944 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:24.690502 7944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "9cb8165d6e7b4a02872912df28ae1a9d"
format_stamp: "Formatted at 2026-05-01 02:09:24 on dist-test-slave-2j90"
server_key: "cdf1598885d08790a39fa730c90c53b6"
server_key_iv: "1e9d6ffb6728c472cef19407c2304ebc"
server_key_version: "encryptionkey@0"
I20260501 02:09:24.690580 7944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:24.721524 7944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:24.724066 7944 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:24.724222 7944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:24.731006 7944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40747
I20260501 02:09:24.731024 8011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40747 every 8 connection(s)
I20260501 02:09:24.731911 7944 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:24.734519 8012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:24.739429 8012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d: Bootstrap starting.
I20260501 02:09:24.739877 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 7944
I20260501 02:09:24.740008 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:24.740245 18819 external_mini_cluster.cc:1468] Setting key e7db73a2affaadba89b58d1ae326799c
I20260501 02:09:24.741696 8012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:24.742393 8012 log.cc:826] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:24.744093 8012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d: No bootstrap required, opened a new log
I20260501 02:09:24.746786 8015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:24.741217 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40565 (local address 127.0.0.1:40747)
0501 02:09:24.741557 (+ 340us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:24.741565 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:24.741594 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:24.742157 (+ 563us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:24.742175 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:24.742431 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:24.742692 (+ 261us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:24.743152 (+ 460us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:24.743897 (+ 745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:24.744767 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:24.744978 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:24.745457 (+ 479us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:24.745473 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:24.745482 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:24.745805 (+ 323us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:24.745834 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:24.745848 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:24.745925 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:24.746125 (+ 200us) server_negotiation.cc:300] Negotiation successful
0501 02:09:24.746262 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":115,"threads_started":1}
I20260501 02:09:24.746904 8012 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260501 02:09:24.747115 8012 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:24.747184 8012 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb8165d6e7b4a02872912df28ae1a9d, State: Initialized, Role: FOLLOWER
I20260501 02:09:24.747587 8012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260501 02:09:24.747737 8012 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:24.747820 8012 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:24.747925 8012 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:24.748850 8012 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260501 02:09:24.749171 8012 leader_election.cc:304] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb8165d6e7b4a02872912df28ae1a9d; no voters:
I20260501 02:09:24.749284 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:24.749470 8012 leader_election.cc:290] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:24.749588 8017 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:24.749797 8017 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [term 1 LEADER]: Becoming Leader. State: Replica: 9cb8165d6e7b4a02872912df28ae1a9d, State: Running, Role: LEADER
I20260501 02:09:24.750164 8017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } }
I20260501 02:09:24.750499 8012 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:24.751272 8017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb8165d6e7b4a02872912df28ae1a9d. Latest consensus state: current_term: 1 leader_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } } }
I20260501 02:09:24.751377 8017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:24.751672 8018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb8165d6e7b4a02872912df28ae1a9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40747 } } }
I20260501 02:09:24.751775 8018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:24.752131 8029 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:24.754984 8029 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:24.756119 8015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:24.752398 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40539 (local address 127.0.0.1:40747)
0501 02:09:24.752535 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:24.752539 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:24.752552 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:09:24.752583 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:24.752585 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:24.752631 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:24.752709 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:24.753157 (+ 448us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:24.753858 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:24.754681 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:24.754869 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:24.755664 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:24.755678 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:24.755681 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:24.755692 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:24.755717 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:24.755730 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:24.755776 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:24.755943 (+ 167us) server_negotiation.cc:300] Negotiation successful
0501 02:09:24.755984 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260501 02:09:24.760051 8029 catalog_manager.cc:1357] Generated new cluster ID: b7fa832217af450ca7db9f69061f6cbe
I20260501 02:09:24.760111 8029 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:24.766043 8029 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:24.766738 8029 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:24.779227 8029 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9cb8165d6e7b4a02872912df28ae1a9d: Generated new TSK 0
I20260501 02:09:24.779731 8029 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:24.788555 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 7944
2026-05-01T02:09:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:24 dist-test-slave-2j90 krb5kdc[8043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:24 dist-test-slave-2j90 krb5kdc[8043](info): set up 2 sockets
May 01 02:09:24 dist-test-slave-2j90 krb5kdc[8043](info): commencing operation
krb5kdc: starting...
W20260501 02:09:26.766517 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:26 dist-test-slave-2j90 krb5kdc[8043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:26.898869 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:26.995554 8059 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:26.995738 8059 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:26.995820 8059 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:26.999078 8059 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:26.999127 8059 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:26.999172 8059 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:26.999212 8059 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:26.999264 8059 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:27.003422 8059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:27.004292 8059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:27.005048 8059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:27.010073 8067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:27.010068 8065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:27.010068 8064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:27.010197 8059 server_base.cc:1061] running on GCE node
I20260501 02:09:27.010689 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:27.011416 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:27.012562 8059 hybrid_clock.cc:648] HybridClock initialized: now 1777601367012540 us; error 33 us; skew 500 ppm
May 01 02:09:27 dist-test-slave-2j90 krb5kdc[8043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:27.015054 8059 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:27.015990 8059 webserver.cc:492] Webserver started at http://127.0.0.1:46269/ using document root <none> and password file <none>
I20260501 02:09:27.016466 8059 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:27.016532 8059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:27.016723 8059 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:27.018215 8059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "a1f870cc9c89499b9881705534c8dec4"
format_stamp: "Formatted at 2026-05-01 02:09:27 on dist-test-slave-2j90"
server_key: "6023a68a486fab396eda8bb59ffe47bd"
server_key_iv: "77c3afce8a89a020fcb46386582ad797"
server_key_version: "encryptionkey@0"
I20260501 02:09:27.018657 8059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "a1f870cc9c89499b9881705534c8dec4"
format_stamp: "Formatted at 2026-05-01 02:09:27 on dist-test-slave-2j90"
server_key: "6023a68a486fab396eda8bb59ffe47bd"
server_key_iv: "77c3afce8a89a020fcb46386582ad797"
server_key_version: "encryptionkey@0"
I20260501 02:09:27.021453 8059 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:27.023379 8074 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:27.024253 8059 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:27.024374 8059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "a1f870cc9c89499b9881705534c8dec4"
format_stamp: "Formatted at 2026-05-01 02:09:27 on dist-test-slave-2j90"
server_key: "6023a68a486fab396eda8bb59ffe47bd"
server_key_iv: "77c3afce8a89a020fcb46386582ad797"
server_key_version: "encryptionkey@0"
I20260501 02:09:27.024478 8059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:27.028541 8059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:27.034222 8059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:27.034391 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:27.041069 8059 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42853
I20260501 02:09:27.041078 8126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42853 every 8 connection(s)
I20260501 02:09:27.041925 8059 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:27.043236 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8059
I20260501 02:09:27.043363 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:27.043655 18819 external_mini_cluster.cc:1468] Setting key 4a098ca06245811344f0a19fb5d46d97
I20260501 02:09:27.044788 8127 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:27.048182 8130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:27.044760 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37865 (local address 127.0.0.1:42853)
0501 02:09:27.045146 (+ 386us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:27.045153 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:27.045182 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:27.045775 (+ 593us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:27.045790 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:27.046049 (+ 259us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:27.046276 (+ 227us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:27.047242 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:27.047246 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:27.047254 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:27.047261 (+ 7us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:27.047281 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:27.047291 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:27.047420 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:27.047527 (+ 107us) server_negotiation.cc:300] Negotiation successful
0501 02:09:27.047646 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260501 02:09:27.049847 8127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4: Bootstrap starting.
I20260501 02:09:27.050626 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:27.052429 8127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:27.053252 8127 log.cc:826] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:27.054352 8130 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:27.052697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58241 (local address 127.0.0.1:42853)
0501 02:09:27.052829 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:27.052835 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:27.053003 (+ 168us) server_negotiation.cc:408] Connection header received
0501 02:09:27.053094 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:27.053097 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:27.053133 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:27.053207 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:27.053942 (+ 735us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:27.053945 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:27.053947 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:27.054003 (+ 56us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:27.054035 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:27.054046 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:27.054146 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:27.054223 (+ 77us) server_negotiation.cc:300] Negotiation successful
0501 02:09:27.054250 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:09:27.055264 8127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4: No bootstrap required, opened a new log
I20260501 02:09:27.057945 8127 raft_consensus.cc:359] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260501 02:09:27.058156 8127 raft_consensus.cc:385] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:27.058188 8127 raft_consensus.cc:740] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f870cc9c89499b9881705534c8dec4, State: Initialized, Role: FOLLOWER
I20260501 02:09:27.058560 8127 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260501 02:09:27.058691 8127 raft_consensus.cc:399] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:27.058763 8127 raft_consensus.cc:493] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:27.058861 8127 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:27.059644 8127 raft_consensus.cc:515] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260501 02:09:27.059948 8127 leader_election.cc:304] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f870cc9c89499b9881705534c8dec4; no voters:
I20260501 02:09:27.060215 8127 leader_election.cc:290] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:27.060345 8137 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:27.060601 8137 raft_consensus.cc:697] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [term 1 LEADER]: Becoming Leader. State: Replica: a1f870cc9c89499b9881705534c8dec4, State: Running, Role: LEADER
I20260501 02:09:27.060883 8137 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } }
I20260501 02:09:27.061225 8127 sys_catalog.cc:565] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:27.062533 8138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f870cc9c89499b9881705534c8dec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } } }
I20260501 02:09:27.062601 8139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f870cc9c89499b9881705534c8dec4. Latest consensus state: current_term: 1 leader_uuid: "a1f870cc9c89499b9881705534c8dec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f870cc9c89499b9881705534c8dec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42853 } } }
I20260501 02:09:27.062673 8138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:27.062693 8139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:27.063030 8146 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:27.066130 8146 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:27.071135 8146 catalog_manager.cc:1357] Generated new cluster ID: f2f270a98a724ada882598c5b59a7d72
I20260501 02:09:27.071223 8146 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:27.082748 8146 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:27.083010 8146 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:27.092788 8146 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a1f870cc9c89499b9881705534c8dec4: Generated new TSK 0
I20260501 02:09:27.093359 8146 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:27.101634 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8059
2026-05-01T02:09:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:27 dist-test-slave-2j90 krb5kdc[8158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:27 dist-test-slave-2j90 krb5kdc[8158](info): set up 2 sockets
May 01 02:09:27 dist-test-slave-2j90 krb5kdc[8158](info): commencing operation
krb5kdc: starting...
W20260501 02:09:29.090538 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:29.222754 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:44821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:29.320128 8174 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:29.320318 8174 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:29.320365 8174 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:29.323653 8174 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:29.323733 8174 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:29.323776 8174 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:29.323856 8174 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:29.323897 8174 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:29.328256 8174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:29.329114 8174 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:29.329821 8174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:29.334877 8179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:29.334836 8182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:29.334842 8180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:29.335129 8174 server_base.cc:1061] running on GCE node
I20260501 02:09:29.335537 8174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:29.336355 8174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:29.337507 8174 hybrid_clock.cc:648] HybridClock initialized: now 1777601369337489 us; error 28 us; skew 500 ppm
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:29.340126 8174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:29.341022 8174 webserver.cc:492] Webserver started at http://127.0.0.1:42249/ using document root <none> and password file <none>
I20260501 02:09:29.341503 8174 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:29.341552 8174 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:29.341696 8174 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:29.343271 8174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "e3fd75cdc6854b60893acd96b1f33929"
format_stamp: "Formatted at 2026-05-01 02:09:29 on dist-test-slave-2j90"
server_key: "2cca7af3dccfc790b883181d9bc6faf7"
server_key_iv: "0493c5b72b3dca18d9192ac0170809cc"
server_key_version: "encryptionkey@0"
I20260501 02:09:29.343694 8174 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "e3fd75cdc6854b60893acd96b1f33929"
format_stamp: "Formatted at 2026-05-01 02:09:29 on dist-test-slave-2j90"
server_key: "2cca7af3dccfc790b883181d9bc6faf7"
server_key_iv: "0493c5b72b3dca18d9192ac0170809cc"
server_key_version: "encryptionkey@0"
I20260501 02:09:29.346709 8174 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260501 02:09:29.348750 8189 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:29.349609 8174 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:29.349721 8174 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "e3fd75cdc6854b60893acd96b1f33929"
format_stamp: "Formatted at 2026-05-01 02:09:29 on dist-test-slave-2j90"
server_key: "2cca7af3dccfc790b883181d9bc6faf7"
server_key_iv: "0493c5b72b3dca18d9192ac0170809cc"
server_key_version: "encryptionkey@0"
I20260501 02:09:29.349805 8174 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:29.357929 8174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:29.360196 8174 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:29.360399 8174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:29.367007 8174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45863
I20260501 02:09:29.367017 8241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45863 every 8 connection(s)
I20260501 02:09:29.367934 8174 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:29.370424 8242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:29.375387 8242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929: Bootstrap starting.
I20260501 02:09:29.377410 8242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:29.377434 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8174
I20260501 02:09:29.377584 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:29.377823 18819 external_mini_cluster.cc:1468] Setting key 06e050d9f6e5edba92a93237b1ecd0dd
I20260501 02:09:29.378268 8242 log.cc:826] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:29.380254 8242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929: No bootstrap required, opened a new log
I20260501 02:09:29.382967 8242 raft_consensus.cc:359] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260501 02:09:29.383121 8242 raft_consensus.cc:385] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:29.383175 8242 raft_consensus.cc:740] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fd75cdc6854b60893acd96b1f33929, State: Initialized, Role: FOLLOWER
I20260501 02:09:29.383581 8242 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260501 02:09:29.383756 8242 raft_consensus.cc:399] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:29.383867 8242 raft_consensus.cc:493] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:29.383975 8242 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:29.384912 8242 raft_consensus.cc:515] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260501 02:09:29.385221 8242 leader_election.cc:304] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3fd75cdc6854b60893acd96b1f33929; no voters:
I20260501 02:09:29.385509 8242 leader_election.cc:290] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:29.385578 8247 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 1 FOLLOWER]: Leader election won for term 1
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260501 02:09:29.385756 8247 raft_consensus.cc:697] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [term 1 LEADER]: Becoming Leader. State: Replica: e3fd75cdc6854b60893acd96b1f33929, State: Running, Role: LEADER
I20260501 02:09:29.386104 8247 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } }
I20260501 02:09:29.386632 8242 sys_catalog.cc:565] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:29.387871 8249 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3fd75cdc6854b60893acd96b1f33929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260501 02:09:29.387997 8249 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:29.388417 8256 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:29.389348 8248 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3fd75cdc6854b60893acd96b1f33929. Latest consensus state: current_term: 1 leader_uuid: "e3fd75cdc6854b60893acd96b1f33929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd75cdc6854b60893acd96b1f33929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45863 } } }
I20260501 02:09:29.389458 8248 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:29.391489 8256 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:29.392928 8246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:29.378842 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:56257 (local address 127.0.0.1:45863)
0501 02:09:29.379249 (+ 407us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:29.379259 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:29.379294 (+ 35us) server_negotiation.cc:408] Connection header received
0501 02:09:29.379941 (+ 647us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:29.379974 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:29.380291 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:29.380616 (+ 325us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:29.381475 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:29.382613 (+ 1138us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:29.383360 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:29.383644 (+ 284us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:29.386357 (+ 2713us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:29.386377 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:29.386391 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:29.386429 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:29.389929 (+ 3500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:29.390576 (+ 647us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:29.390582 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:29.390587 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:29.390665 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:29.390974 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:29.390977 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:29.390980 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:29.391249 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:29.391443 (+ 194us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:29.392262 (+ 819us) server_negotiation.cc:300] Negotiation successful
0501 02:09:29.392418 (+ 156us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260501 02:09:29.395419 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:29.396939 8256 catalog_manager.cc:1357] Generated new cluster ID: 2aa8a363115b4107a5c2fe314909ef99
I20260501 02:09:29.396996 8256 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:29.405490 8256 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:29.406495 8256 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:29.409886 8246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:29.402318 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:39403 (local address 127.0.0.1:45863)
0501 02:09:29.402430 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:29.402433 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:29.402743 (+ 310us) server_negotiation.cc:408] Connection header received
0501 02:09:29.402881 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:29.402889 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:29.402929 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:29.403007 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:29.403752 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:29.404231 (+ 479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:29.405098 (+ 867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:29.405283 (+ 185us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:29.406985 (+ 1702us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:29.407002 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:29.407005 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:29.407031 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:29.408366 (+ 1335us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:29.408791 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:29.408799 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:29.408800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:29.408840 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:29.409104 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:29.409107 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:29.409109 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:29.409232 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:29.409378 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:29.409631 (+ 253us) server_negotiation.cc:300] Negotiation successful
0501 02:09:29.409714 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:09:29.423045 8256 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e3fd75cdc6854b60893acd96b1f33929: Generated new TSK 0
I20260501 02:09:29.423642 8256 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:29.430737 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8174
W20260501 02:09:29.436335 8156 connection.cc:570] client connection to 127.0.0.1:45863 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45863): Connection reset by peer (error 104)
2026-05-01T02:09:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8273](info): set up 2 sockets
May 01 02:09:29 dist-test-slave-2j90 krb5kdc[8273](info): commencing operation
krb5kdc: starting...
W20260501 02:09:31.402503 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:31 dist-test-slave-2j90 krb5kdc[8273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:31.532663 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35749
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:31.627506 8289 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:31.627699 8289 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:31.627799 8289 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:31.630970 8289 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:31.631024 8289 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:31.631083 8289 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:31.631122 8289 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:31.631170 8289 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:31.635249 8289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:31.636168 8289 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:31.636926 8289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:31.642078 8294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:31.642122 8295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:31.642086 8297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:31.642298 8289 server_base.cc:1061] running on GCE node
I20260501 02:09:31.642727 8289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:31.643524 8289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:31.644687 8289 hybrid_clock.cc:648] HybridClock initialized: now 1777601371644662 us; error 38 us; skew 500 ppm
May 01 02:09:31 dist-test-slave-2j90 krb5kdc[8273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:31.647169 8289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:31.648088 8289 webserver.cc:492] Webserver started at http://127.0.0.1:39783/ using document root <none> and password file <none>
I20260501 02:09:31.648552 8289 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:31.648594 8289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:31.648734 8289 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:31.650209 8289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "b62e843408c54a8388f87d33f1ed0b00"
format_stamp: "Formatted at 2026-05-01 02:09:31 on dist-test-slave-2j90"
server_key: "0f54fdbbf314389138ba85c738387596"
server_key_iv: "bcbed31cf340540c7759a287d74faf67"
server_key_version: "encryptionkey@0"
I20260501 02:09:31.650609 8289 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "b62e843408c54a8388f87d33f1ed0b00"
format_stamp: "Formatted at 2026-05-01 02:09:31 on dist-test-slave-2j90"
server_key: "0f54fdbbf314389138ba85c738387596"
server_key_iv: "bcbed31cf340540c7759a287d74faf67"
server_key_version: "encryptionkey@0"
I20260501 02:09:31.653343 8289 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:31.655241 8304 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:31.656100 8289 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260501 02:09:31.656208 8289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "b62e843408c54a8388f87d33f1ed0b00"
format_stamp: "Formatted at 2026-05-01 02:09:31 on dist-test-slave-2j90"
server_key: "0f54fdbbf314389138ba85c738387596"
server_key_iv: "bcbed31cf340540c7759a287d74faf67"
server_key_version: "encryptionkey@0"
I20260501 02:09:31.656292 8289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:31.665752 8289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:31.667919 8289 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:31.668085 8289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:31.674676 8289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35749
I20260501 02:09:31.674691 8356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:35749 every 8 connection(s)
I20260501 02:09:31.675558 8289 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:31.677081 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8289
I20260501 02:09:31.677218 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:31.677457 18819 external_mini_cluster.cc:1468] Setting key 257ed791d93e12bb1290afed12125fbc
I20260501 02:09:31.678218 8357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:31.684046 8357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00: Bootstrap starting.
I20260501 02:09:31.684509 8360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:31.678531 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:50897 (local address 127.0.0.1:35749)
0501 02:09:31.678880 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:31.678888 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:31.678917 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:31.679502 (+ 585us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:31.679517 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:31.679766 (+ 249us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:31.680010 (+ 244us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:31.680652 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:31.681378 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:31.682248 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:31.682440 (+ 192us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:31.683058 (+ 618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:31.683095 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:31.683104 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:31.683468 (+ 364us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:31.683497 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:31.683511 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:31.683609 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:31.683819 (+ 210us) server_negotiation.cc:300] Negotiation successful
0501 02:09:31.683929 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":123,"threads_started":1}
I20260501 02:09:31.686311 8357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:31.687021 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:31.687073 8357 log.cc:826] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:31.688879 8357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00: No bootstrap required, opened a new log
I20260501 02:09:31.691433 8357 raft_consensus.cc:359] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260501 02:09:31.691620 8357 raft_consensus.cc:385] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:31.691675 8357 raft_consensus.cc:740] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62e843408c54a8388f87d33f1ed0b00, State: Initialized, Role: FOLLOWER
I20260501 02:09:31.692077 8357 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260501 02:09:31.692217 8357 raft_consensus.cc:399] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:31.692327 8357 raft_consensus.cc:493] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:31.692431 8360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:31.689129 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:37853 (local address 127.0.0.1:35749)
0501 02:09:31.689254 (+ 125us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:31.689263 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:31.689452 (+ 189us) server_negotiation.cc:408] Connection header received
0501 02:09:31.689573 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:31.689577 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:31.689631 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:31.689708 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:31.690154 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:31.690585 (+ 431us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:31.691173 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:31.691661 (+ 488us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:31.691873 (+ 212us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:31.691888 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:31.691891 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:31.691903 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:31.691928 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:31.691939 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:31.692032 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:31.692170 (+ 138us) server_negotiation.cc:300] Negotiation successful
0501 02:09:31.692195 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:09:31.692448 8357 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:31.693849 8357 raft_consensus.cc:515] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260501 02:09:31.694212 8357 leader_election.cc:304] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62e843408c54a8388f87d33f1ed0b00; no voters:
I20260501 02:09:31.694478 8357 leader_election.cc:290] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:31.694612 8367 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:31.694820 8367 raft_consensus.cc:697] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [term 1 LEADER]: Becoming Leader. State: Replica: b62e843408c54a8388f87d33f1ed0b00, State: Running, Role: LEADER
I20260501 02:09:31.695099 8367 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } }
I20260501 02:09:31.695575 8357 sys_catalog.cc:565] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:31.697217 8369 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62e843408c54a8388f87d33f1ed0b00. Latest consensus state: current_term: 1 leader_uuid: "b62e843408c54a8388f87d33f1ed0b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } } }
I20260501 02:09:31.697283 8368 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b62e843408c54a8388f87d33f1ed0b00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62e843408c54a8388f87d33f1ed0b00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35749 } } }
I20260501 02:09:31.697345 8369 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:31.697395 8368 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:31.698102 8376 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:31.700969 8376 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:31.706102 8376 catalog_manager.cc:1357] Generated new cluster ID: 7838cdcfbbf140899117f20e115ab233
I20260501 02:09:31.706157 8376 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:31.723884 8376 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:31.724668 8376 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:31.729615 8376 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b62e843408c54a8388f87d33f1ed0b00: Generated new TSK 0
I20260501 02:09:31.730111 8376 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:31.734969 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8289
W20260501 02:09:31.740477 8271 connection.cc:570] client connection to 127.0.0.1:35749 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35749): Connection reset by peer (error 104)
2026-05-01T02:09:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:31 dist-test-slave-2j90 krb5kdc[8388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:31 dist-test-slave-2j90 krb5kdc[8388](info): set up 2 sockets
May 01 02:09:31 dist-test-slave-2j90 krb5kdc[8388](info): commencing operation
krb5kdc: starting...
W20260501 02:09:33.718626 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.964s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:33 dist-test-slave-2j90 krb5kdc[8388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:33.850876 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:33.947090 8404 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:33.947275 8404 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:33.947364 8404 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:33.950531 8404 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:33.950582 8404 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:33.950604 8404 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:33.950661 8404 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:33.950698 8404 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:33.954732 8404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:33.955606 8404 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:33.956339 8404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:33.961489 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:33.961544 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:33.961498 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:33.961859 8404 server_base.cc:1061] running on GCE node
I20260501 02:09:33.962329 8404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:33.963210 8404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:33.964404 8404 hybrid_clock.cc:648] HybridClock initialized: now 1777601373964385 us; error 45 us; skew 500 ppm
May 01 02:09:33 dist-test-slave-2j90 krb5kdc[8388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:33.967092 8404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:33.968144 8404 webserver.cc:492] Webserver started at http://127.0.0.1:41941/ using document root <none> and password file <none>
I20260501 02:09:33.968694 8404 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:33.968766 8404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:33.968971 8404 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:33.970610 8404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03"
format_stamp: "Formatted at 2026-05-01 02:09:33 on dist-test-slave-2j90"
server_key: "1c4812893d6a1bd8c224ea123c67f56c"
server_key_iv: "e73d8a6f0802d4e5d30dc153aafcf470"
server_key_version: "encryptionkey@0"
I20260501 02:09:33.971079 8404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03"
format_stamp: "Formatted at 2026-05-01 02:09:33 on dist-test-slave-2j90"
server_key: "1c4812893d6a1bd8c224ea123c67f56c"
server_key_iv: "e73d8a6f0802d4e5d30dc153aafcf470"
server_key_version: "encryptionkey@0"
I20260501 02:09:33.974125 8404 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:09:33.976022 8419 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:33.976924 8404 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:33.977051 8404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03"
format_stamp: "Formatted at 2026-05-01 02:09:33 on dist-test-slave-2j90"
server_key: "1c4812893d6a1bd8c224ea123c67f56c"
server_key_iv: "e73d8a6f0802d4e5d30dc153aafcf470"
server_key_version: "encryptionkey@0"
I20260501 02:09:33.977142 8404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:33.981314 8404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:33.986383 8404 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:33.986559 8404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:33.993479 8404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40703
I20260501 02:09:33.993479 8471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:40703 every 8 connection(s)
I20260501 02:09:33.994459 8404 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:33.995266 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8404
I20260501 02:09:33.995412 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:33.995678 18819 external_mini_cluster.cc:1468] Setting key 366238a3174031f2e80ec038164ddf46
I20260501 02:09:33.997411 8472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:34.000072 8475 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:33.996739 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:55885 (local address 127.0.0.1:40703)
0501 02:09:33.997075 (+ 336us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:33.997083 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:33.997111 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:09:33.997768 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:33.997784 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:33.998052 (+ 268us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:33.998285 (+ 233us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:33.999157 (+ 872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:33.999161 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:33.999169 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:33.999178 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:33.999199 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:33.999209 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:33.999307 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:33.999449 (+ 142us) server_negotiation.cc:300] Negotiation successful
0501 02:09:33.999567 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":113,"threads_started":1}
I20260501 02:09:34.002642 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:34.002836 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03: Bootstrap starting.
I20260501 02:09:34.005357 8472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:34.006217 8472 log.cc:826] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:34.006358 8475 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:34.004784 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:51329 (local address 127.0.0.1:40703)
0501 02:09:34.004904 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:34.004907 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:34.005037 (+ 130us) server_negotiation.cc:408] Connection header received
0501 02:09:34.005139 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:34.005142 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:34.005168 (+ 26us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:34.005231 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:34.005868 (+ 637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:34.005871 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:34.005873 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:34.005882 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:34.005906 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:34.005915 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:34.006051 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:34.006149 (+ 98us) server_negotiation.cc:300] Negotiation successful
0501 02:09:34.006175 (+ 26us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:09:34.007956 8472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03: No bootstrap required, opened a new log
I20260501 02:09:34.010522 8472 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } }
I20260501 02:09:34.010720 8472 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:34.010802 8472 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1478e8baedd04e8cb4aa6ae5b00a6c03, State: Initialized, Role: FOLLOWER
I20260501 02:09:34.011307 8472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } }
I20260501 02:09:34.011508 8472 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:34.011585 8472 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:34.011708 8472 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:34.012446 8472 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } }
I20260501 02:09:34.012733 8472 leader_election.cc:304] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1478e8baedd04e8cb4aa6ae5b00a6c03; no voters:
I20260501 02:09:34.012969 8472 leader_election.cc:290] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:34.013073 8482 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:34.013303 8482 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [term 1 LEADER]: Becoming Leader. State: Replica: 1478e8baedd04e8cb4aa6ae5b00a6c03, State: Running, Role: LEADER
I20260501 02:09:34.013618 8482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } }
I20260501 02:09:34.014135 8472 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:34.015161 8483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } } }
I20260501 02:09:34.015204 8484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1478e8baedd04e8cb4aa6ae5b00a6c03. Latest consensus state: current_term: 1 leader_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1478e8baedd04e8cb4aa6ae5b00a6c03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40703 } } }
I20260501 02:09:34.015295 8483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:34.015338 8484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:34.015712 8489 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:34.018239 8489 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:34.022984 8489 catalog_manager.cc:1357] Generated new cluster ID: 81f4b04480dd469c80b632f5f3ea444c
I20260501 02:09:34.023043 8489 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:34.070207 8489 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:34.070547 8489 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:34.078778 8489 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1478e8baedd04e8cb4aa6ae5b00a6c03: Generated new TSK 0
I20260501 02:09:34.079316 8489 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:34.085176 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8404
2026-05-01T02:09:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:34 dist-test-slave-2j90 krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:34 dist-test-slave-2j90 krb5kdc[8503](info): set up 2 sockets
May 01 02:09:34 dist-test-slave-2j90 krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260501 02:09:36.070647 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.965s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:36.209379 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:36.309497 8519 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:36.309752 8519 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:36.309845 8519 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:36.313799 8519 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:36.313886 8519 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:36.313928 8519 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:36.314005 8519 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:36.314050 8519 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:36.318331 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:36.319319 8519 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:36.320127 8519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:36.325409 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:36.325467 8519 server_base.cc:1061] running on GCE node
W20260501 02:09:36.325628 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:36.325490 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:36.326270 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:36.327126 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:36.328275 8519 hybrid_clock.cc:648] HybridClock initialized: now 1777601376328249 us; error 38 us; skew 500 ppm
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:36.331037 8519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:36.331939 8519 webserver.cc:492] Webserver started at http://127.0.0.1:33453/ using document root <none> and password file <none>
I20260501 02:09:36.332409 8519 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:36.332453 8519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:36.332597 8519 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:36.334120 8519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "342e0e66ae0049df830cf1b4598bb476"
format_stamp: "Formatted at 2026-05-01 02:09:36 on dist-test-slave-2j90"
server_key: "40b85c751e0fddc6127e1e204a6df542"
server_key_iv: "ce55fe15addaf47109d76f6024dc0f50"
server_key_version: "encryptionkey@0"
I20260501 02:09:36.334553 8519 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "342e0e66ae0049df830cf1b4598bb476"
format_stamp: "Formatted at 2026-05-01 02:09:36 on dist-test-slave-2j90"
server_key: "40b85c751e0fddc6127e1e204a6df542"
server_key_iv: "ce55fe15addaf47109d76f6024dc0f50"
server_key_version: "encryptionkey@0"
I20260501 02:09:36.337482 8519 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260501 02:09:36.339448 8534 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:36.340334 8519 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:36.340459 8519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "342e0e66ae0049df830cf1b4598bb476"
format_stamp: "Formatted at 2026-05-01 02:09:36 on dist-test-slave-2j90"
server_key: "40b85c751e0fddc6127e1e204a6df542"
server_key_iv: "ce55fe15addaf47109d76f6024dc0f50"
server_key_version: "encryptionkey@0"
I20260501 02:09:36.340551 8519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:36.351430 8519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:36.354099 8519 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:36.354285 8519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:36.360812 8519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36237
I20260501 02:09:36.360826 8586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36237 every 8 connection(s)
I20260501 02:09:36.361732 8519 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:36.364280 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8519
I20260501 02:09:36.364428 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:36.364494 8587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:36.364712 18819 external_mini_cluster.cc:1468] Setting key 6a92765f3425f7ec3854340a6047df68
I20260501 02:09:36.370128 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476: Bootstrap starting.
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260501 02:09:36.372160 8587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:36.372823 8587 log.cc:826] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:36.374784 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476: No bootstrap required, opened a new log
I20260501 02:09:36.377269 8590 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:36.365814 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:59563 (local address 127.0.0.1:36237)
0501 02:09:36.366198 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:36.366206 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:36.366235 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:36.367349 (+ 1114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:36.367367 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:36.367628 (+ 261us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:36.367920 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:36.368829 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:36.369543 (+ 714us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:36.370292 (+ 749us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:36.370567 (+ 275us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:36.372673 (+ 2106us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:36.372697 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:36.372714 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:36.372740 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:36.374764 (+ 2024us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:36.375247 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:36.375252 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:36.375257 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:36.375340 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:36.375625 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:36.375628 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:36.375630 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:36.375993 (+ 363us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:36.376172 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:36.376433 (+ 261us) server_negotiation.cc:300] Negotiation successful
0501 02:09:36.376616 (+ 183us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":135,"threads_started":1}
I20260501 02:09:36.377811 8587 raft_consensus.cc:359] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260501 02:09:36.378031 8587 raft_consensus.cc:385] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:36.378099 8587 raft_consensus.cc:740] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342e0e66ae0049df830cf1b4598bb476, State: Initialized, Role: FOLLOWER
I20260501 02:09:36.378495 8587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260501 02:09:36.378612 8587 raft_consensus.cc:399] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:36.378701 8587 raft_consensus.cc:493] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:36.378821 8587 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:36.379771 8587 raft_consensus.cc:515] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260501 02:09:36.379915 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:36.380121 8587 leader_election.cc:304] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342e0e66ae0049df830cf1b4598bb476; no voters:
I20260501 02:09:36.380414 8587 leader_election.cc:290] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:36.380779 8592 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:36.381001 8592 raft_consensus.cc:697] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [term 1 LEADER]: Becoming Leader. State: Replica: 342e0e66ae0049df830cf1b4598bb476, State: Running, Role: LEADER
I20260501 02:09:36.381296 8592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260501 02:09:36.381508 8587 sys_catalog.cc:565] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:36.382719 8594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342e0e66ae0049df830cf1b4598bb476. Latest consensus state: current_term: 1 leader_uuid: "342e0e66ae0049df830cf1b4598bb476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } } }
I20260501 02:09:36.383008 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342e0e66ae0049df830cf1b4598bb476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e0e66ae0049df830cf1b4598bb476" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } } }
I20260501 02:09:36.383108 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:36.383347 8594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:36.384032 8606 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:36.387137 8606 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:36.389523 8590 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:36.381921 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:58871 (local address 127.0.0.1:36237)
0501 02:09:36.382059 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:36.382063 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:36.382613 (+ 550us) server_negotiation.cc:408] Connection header received
0501 02:09:36.382800 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:36.382804 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:36.382852 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:36.382911 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:36.383987 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:36.384606 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:36.385185 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:36.385403 (+ 218us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:36.386461 (+ 1058us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:36.386478 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:36.386483 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:36.386510 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:36.387890 (+ 1380us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:36.388324 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:36.388330 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:36.388333 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:36.388387 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:36.388728 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:36.388738 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:36.388740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:36.388870 (+ 130us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:36.389030 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:36.389276 (+ 246us) server_negotiation.cc:300] Negotiation successful
0501 02:09:36.389357 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260501 02:09:36.392163 8606 catalog_manager.cc:1357] Generated new cluster ID: 6e2f64f225c64daf8f2f04846811333e
I20260501 02:09:36.392227 8606 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:36.419349 8606 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:36.420501 8606 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:36.435967 8606 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 342e0e66ae0049df830cf1b4598bb476: Generated new TSK 0
I20260501 02:09:36.436592 8606 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:36.442010 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8519
W20260501 02:09:36.447688 8501 connection.cc:570] client connection to 127.0.0.1:36237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36237): Connection reset by peer (error 104)
2026-05-01T02:09:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8618](info): set up 2 sockets
May 01 02:09:36 dist-test-slave-2j90 krb5kdc[8618](info): commencing operation
krb5kdc: starting...
W20260501 02:09:38.430459 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:38 dist-test-slave-2j90 krb5kdc[8618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:38.561338 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:38.658286 8634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:38.658511 8634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:38.658594 8634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:38.661522 8634 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:38.661612 8634 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:38.661681 8634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:38.661726 8634 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:38.661765 8634 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:38.665787 8634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:38.666724 8634 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:38.667438 8634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:38.672451 8640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:38.672569 8639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:38.672559 8634 server_base.cc:1061] running on GCE node
W20260501 02:09:38.672472 8642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:38.673174 8634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:38.674036 8634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:38.675215 8634 hybrid_clock.cc:648] HybridClock initialized: now 1777601378675202 us; error 43 us; skew 500 ppm
May 01 02:09:38 dist-test-slave-2j90 krb5kdc[8618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:38.677753 8634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:38.678845 8634 webserver.cc:492] Webserver started at http://127.0.0.1:44349/ using document root <none> and password file <none>
I20260501 02:09:38.679381 8634 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:38.679456 8634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:38.679672 8634 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:38.681234 8634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "0c7328f8134d42e5ae6c5475768ae063"
format_stamp: "Formatted at 2026-05-01 02:09:38 on dist-test-slave-2j90"
server_key: "bb94c11446ee7f56bdd513ec8177f862"
server_key_iv: "2eed65a2398d9ea95c8accac2ed6a662"
server_key_version: "encryptionkey@0"
I20260501 02:09:38.681711 8634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "0c7328f8134d42e5ae6c5475768ae063"
format_stamp: "Formatted at 2026-05-01 02:09:38 on dist-test-slave-2j90"
server_key: "bb94c11446ee7f56bdd513ec8177f862"
server_key_iv: "2eed65a2398d9ea95c8accac2ed6a662"
server_key_version: "encryptionkey@0"
I20260501 02:09:38.685708 8634 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260501 02:09:38.688580 8649 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:38.689920 8634 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260501 02:09:38.690063 8634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "0c7328f8134d42e5ae6c5475768ae063"
format_stamp: "Formatted at 2026-05-01 02:09:38 on dist-test-slave-2j90"
server_key: "bb94c11446ee7f56bdd513ec8177f862"
server_key_iv: "2eed65a2398d9ea95c8accac2ed6a662"
server_key_version: "encryptionkey@0"
I20260501 02:09:38.690171 8634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:38.713176 8634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:38.715678 8634 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:38.715863 8634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:38.722780 8634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42679
I20260501 02:09:38.722797 8701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:42679 every 8 connection(s)
I20260501 02:09:38.723786 8634 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:38.725908 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8634
I20260501 02:09:38.726083 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:38.726334 18819 external_mini_cluster.cc:1468] Setting key 91beeb3e6cc4557c97ff39c6ab5dd248
I20260501 02:09:38.726569 8702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:38.733059 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: Bootstrap starting.
I20260501 02:09:38.733820 8705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:38.727456 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:46855 (local address 127.0.0.1:42679)
0501 02:09:38.727872 (+ 416us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:38.727883 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:38.727919 (+ 36us) server_negotiation.cc:408] Connection header received
0501 02:09:38.728703 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:38.728726 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:38.729109 (+ 383us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:38.729447 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:38.729841 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:38.730928 (+ 1087us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:38.731572 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:38.731834 (+ 262us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:38.732236 (+ 402us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:38.732277 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:38.732289 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:38.732737 (+ 448us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:38.732778 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:38.732796 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:38.732921 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:38.733077 (+ 156us) server_negotiation.cc:300] Negotiation successful
0501 02:09:38.733168 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260501 02:09:38.735385 8702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:38.736195 8702 log.cc:826] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:38.736568 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:38.738204 8702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: No bootstrap required, opened a new log
I20260501 02:09:38.741298 8702 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260501 02:09:38.741500 8702 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:38.741540 8702 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7328f8134d42e5ae6c5475768ae063, State: Initialized, Role: FOLLOWER
I20260501 02:09:38.741931 8702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260501 02:09:38.742103 8702 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:38.742169 8702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:38.742316 8702 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:38.742465 8705 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:38.738732 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:57041 (local address 127.0.0.1:42679)
0501 02:09:38.738833 (+ 101us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:38.738838 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:38.739075 (+ 237us) server_negotiation.cc:408] Connection header received
0501 02:09:38.739183 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:38.739186 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:38.739226 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:38.739305 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:38.739803 (+ 498us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:38.740220 (+ 417us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:38.741026 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:38.741137 (+ 111us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:38.741915 (+ 778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:38.741928 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:38.741930 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:38.741939 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:38.742008 (+ 69us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:38.742020 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:38.742123 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:38.742284 (+ 161us) server_negotiation.cc:300] Negotiation successful
0501 02:09:38.742332 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260501 02:09:38.743722 8702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260501 02:09:38.744014 8702 leader_election.cc:304] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7328f8134d42e5ae6c5475768ae063; no voters:
I20260501 02:09:38.744275 8702 leader_election.cc:290] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:38.744407 8712 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:38.744627 8712 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7328f8134d42e5ae6c5475768ae063, State: Running, Role: LEADER
I20260501 02:09:38.744925 8712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260501 02:09:38.745358 8702 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:38.746448 8713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7328f8134d42e5ae6c5475768ae063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260501 02:09:38.746454 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7328f8134d42e5ae6c5475768ae063. Latest consensus state: current_term: 1 leader_uuid: "0c7328f8134d42e5ae6c5475768ae063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7328f8134d42e5ae6c5475768ae063" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260501 02:09:38.746610 8713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:38.746657 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063 [sys.catalog]: This master's current role is: LEADER
W20260501 02:09:38.749711 8728 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:38.749791 8728 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:38.749869 8721 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:38.750572 8721 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:38.754751 8721 catalog_manager.cc:1357] Generated new cluster ID: 609620f514814779a36e069f20b7350f
I20260501 02:09:38.754808 8721 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:38.768329 8721 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:38.769094 8721 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:38.773531 8721 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0c7328f8134d42e5ae6c5475768ae063: Generated new TSK 0
I20260501 02:09:38.774055 8721 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:38.779888 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8634
W20260501 02:09:38.785392 8616 connection.cc:570] client connection to 127.0.0.1:42679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42679): Connection reset by peer (error 104)
2026-05-01T02:09:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:38 dist-test-slave-2j90 krb5kdc[8733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:38 dist-test-slave-2j90 krb5kdc[8733](info): set up 2 sockets
May 01 02:09:38 dist-test-slave-2j90 krb5kdc[8733](info): commencing operation
krb5kdc: starting...
W20260501 02:09:40.742532 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.943s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:40 dist-test-slave-2j90 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:40.875321 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:40.974050 8749 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:40.974256 8749 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:40.974316 8749 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:40.977483 8749 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:40.977546 8749 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:40.977576 8749 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:40.977597 8749 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:40.977617 8749 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:40.981818 8749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:40.982774 8749 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:40.983532 8749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:40.988660 8757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:40.988665 8754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:40.989010 8749 server_base.cc:1061] running on GCE node
W20260501 02:09:40.988667 8755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:40.989488 8749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:40.990386 8749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:40.991564 8749 hybrid_clock.cc:648] HybridClock initialized: now 1777601380991546 us; error 38 us; skew 500 ppm
May 01 02:09:40 dist-test-slave-2j90 krb5kdc[8733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:40.993947 8749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260501 02:09:40.994951 8749 webserver.cc:492] Webserver started at http://127.0.0.1:36503/ using document root <none> and password file <none>
I20260501 02:09:40.995462 8749 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:40.995535 8749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:40.995746 8749 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:40.997288 8749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "bec18b5eee96497781378a9d5706a1f3"
format_stamp: "Formatted at 2026-05-01 02:09:40 on dist-test-slave-2j90"
server_key: "55dfb460f34820d6fbda4e80da34c543"
server_key_iv: "c40ce556d6376f8345ca6a5422801489"
server_key_version: "encryptionkey@0"
I20260501 02:09:40.997761 8749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "bec18b5eee96497781378a9d5706a1f3"
format_stamp: "Formatted at 2026-05-01 02:09:40 on dist-test-slave-2j90"
server_key: "55dfb460f34820d6fbda4e80da34c543"
server_key_iv: "c40ce556d6376f8345ca6a5422801489"
server_key_version: "encryptionkey@0"
I20260501 02:09:41.000877 8749 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260501 02:09:41.002928 8764 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:41.004153 8749 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:41.004292 8749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "bec18b5eee96497781378a9d5706a1f3"
format_stamp: "Formatted at 2026-05-01 02:09:40 on dist-test-slave-2j90"
server_key: "55dfb460f34820d6fbda4e80da34c543"
server_key_iv: "c40ce556d6376f8345ca6a5422801489"
server_key_version: "encryptionkey@0"
I20260501 02:09:41.004383 8749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:41.008574 8749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:41.013538 8749 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:41.013732 8749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:41.020300 8749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39367
I20260501 02:09:41.020313 8816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39367 every 8 connection(s)
I20260501 02:09:41.021166 8749 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:41.023588 8817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:41.028375 8817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: Bootstrap starting.
I20260501 02:09:41.029822 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8749
I20260501 02:09:41.029999 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:41.030270 18819 external_mini_cluster.cc:1468] Setting key 7ff59e4ad9620afcd1f064aaf01eef69
I20260501 02:09:41.030814 8817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:41.031570 8817 log.cc:826] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:41.033272 8817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: No bootstrap required, opened a new log
I20260501 02:09:41.034345 8820 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:41.031140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:59881 (local address 127.0.0.1:39367)
0501 02:09:41.031507 (+ 367us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:41.031516 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:41.031544 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:09:41.032190 (+ 646us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:41.032213 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:41.032439 (+ 226us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:41.032742 (+ 303us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:41.033425 (+ 683us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:41.033437 (+ 12us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:41.033446 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:41.033454 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:41.033481 (+ 27us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:41.033492 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:41.033590 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:41.033706 (+ 116us) server_negotiation.cc:300] Negotiation successful
0501 02:09:41.033804 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260501 02:09:41.036010 8817 raft_consensus.cc:359] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260501 02:09:41.036204 8817 raft_consensus.cc:385] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:41.036264 8817 raft_consensus.cc:740] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec18b5eee96497781378a9d5706a1f3, State: Initialized, Role: FOLLOWER
I20260501 02:09:41.036677 8817 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260501 02:09:41.036774 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:41.036834 8817 raft_consensus.cc:399] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:41.036895 8817 raft_consensus.cc:493] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:41.036976 8817 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:41.037717 8817 raft_consensus.cc:515] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260501 02:09:41.038077 8817 leader_election.cc:304] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bec18b5eee96497781378a9d5706a1f3; no voters:
I20260501 02:09:41.038367 8817 leader_election.cc:290] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:41.038465 8826 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:41.038645 8826 raft_consensus.cc:697] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [term 1 LEADER]: Becoming Leader. State: Replica: bec18b5eee96497781378a9d5706a1f3, State: Running, Role: LEADER
I20260501 02:09:41.038889 8826 consensus_queue.cc:237] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260501 02:09:41.039397 8817 sys_catalog.cc:565] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:41.039884 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bec18b5eee96497781378a9d5706a1f3. Latest consensus state: current_term: 1 leader_uuid: "bec18b5eee96497781378a9d5706a1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260501 02:09:41.039984 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:41.040639 8827 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bec18b5eee96497781378a9d5706a1f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec18b5eee96497781378a9d5706a1f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260501 02:09:41.040786 8820 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:41.039021 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 10.240.0.95:49601 (local address 127.0.0.1:39367)
0501 02:09:41.039169 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:41.039172 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:41.039184 (+ 12us) server_negotiation.cc:408] Connection header received
0501 02:09:41.039276 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:41.039278 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:41.039303 (+ 25us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:41.039343 (+ 40us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:41.040138 (+ 795us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:41.040143 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:41.040147 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:41.040158 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:41.040189 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:41.040203 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:41.040284 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:41.040596 (+ 312us) server_negotiation.cc:300] Negotiation successful
0501 02:09:41.040635 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260501 02:09:41.040797 8827 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3 [sys.catalog]: This master's current role is: LEADER
W20260501 02:09:41.043905 8842 catalog_manager.cc:1568] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:41.043990 8842 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:41.044064 8831 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:41.044797 8831 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:41.049350 8831 catalog_manager.cc:1357] Generated new cluster ID: 71f329dab83746b887994b04aa24387b
I20260501 02:09:41.049427 8831 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:41.068104 8831 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:41.068359 8831 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:41.080937 8831 catalog_manager.cc:6044] T 00000000000000000000000000000000 P bec18b5eee96497781378a9d5706a1f3: Generated new TSK 0
I20260501 02:09:41.081462 8831 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:41.085506 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8749
2026-05-01T02:09:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:41 dist-test-slave-2j90 krb5kdc[8848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:41 dist-test-slave-2j90 krb5kdc[8848](info): set up 2 sockets
May 01 02:09:41 dist-test-slave-2j90 krb5kdc[8848](info): commencing operation
krb5kdc: starting...
W20260501 02:09:43.042542 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:43.176795 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39797
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:43.272590 8864 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:43.272778 8864 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:43.272823 8864 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:43.276350 8864 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:43.276408 8864 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:43.276432 8864 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:43.276449 8864 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:43.276465 8864 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:43.280745 8864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:39797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:43.281723 8864 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:43.282578 8864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:43.288053 8870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:43.288131 8864 server_base.cc:1061] running on GCE node
W20260501 02:09:43.287943 8869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:43.287945 8872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:43.288739 8864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:43.289623 8864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:43.290797 8864 hybrid_clock.cc:648] HybridClock initialized: now 1777601383290779 us; error 42 us; skew 500 ppm
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601383, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:43.293339 8864 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:43.294304 8864 webserver.cc:492] Webserver started at http://127.18.96.254:33325/ using document root <none> and password file <none>
I20260501 02:09:43.294806 8864 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:43.294852 8864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:43.294998 8864 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:43.296494 8864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "267047fc785e4d8582c185fb64661b4b"
format_stamp: "Formatted at 2026-05-01 02:09:43 on dist-test-slave-2j90"
server_key: "4567bad4d1c05a3ff915152397a4236f"
server_key_iv: "ea28aefef8dfa0ebfa675f6856fa69db"
server_key_version: "encryptionkey@0"
I20260501 02:09:43.296904 8864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "267047fc785e4d8582c185fb64661b4b"
format_stamp: "Formatted at 2026-05-01 02:09:43 on dist-test-slave-2j90"
server_key: "4567bad4d1c05a3ff915152397a4236f"
server_key_iv: "ea28aefef8dfa0ebfa675f6856fa69db"
server_key_version: "encryptionkey@0"
I20260501 02:09:43.299942 8864 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:09:43.301891 8879 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:43.302851 8864 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:43.302970 8864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "267047fc785e4d8582c185fb64661b4b"
format_stamp: "Formatted at 2026-05-01 02:09:43 on dist-test-slave-2j90"
server_key: "4567bad4d1c05a3ff915152397a4236f"
server_key_iv: "ea28aefef8dfa0ebfa675f6856fa69db"
server_key_version: "encryptionkey@0"
I20260501 02:09:43.303064 8864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:43.325407 8864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:43.327703 8864 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:43.327895 8864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:43.334331 8864 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:39797
I20260501 02:09:43.334347 8931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:39797 every 8 connection(s)
I20260501 02:09:43.335218 8864 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:43.337721 8932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:43.341825 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8864
I20260501 02:09:43.341982 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:43.342231 18819 external_mini_cluster.cc:1468] Setting key 6f4d90fefbea7015d33f3f09bd8e0945
I20260501 02:09:43.342725 8932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: Bootstrap starting.
I20260501 02:09:43.344842 8932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:43.345683 8932 log.cc:826] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:43.347544 8932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: No bootstrap required, opened a new log
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:43.350395 8932 raft_consensus.cc:359] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } }
I20260501 02:09:43.350597 8932 raft_consensus.cc:385] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:43.350675 8932 raft_consensus.cc:740] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267047fc785e4d8582c185fb64661b4b, State: Initialized, Role: FOLLOWER
I20260501 02:09:43.351133 8932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } }
I20260501 02:09:43.351285 8932 raft_consensus.cc:399] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:43.351359 8932 raft_consensus.cc:493] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:43.351504 8932 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:43.352262 8932 raft_consensus.cc:515] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } }
I20260501 02:09:43.352558 8932 leader_election.cc:304] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 267047fc785e4d8582c185fb64661b4b; no voters:
I20260501 02:09:43.352816 8932 leader_election.cc:290] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:43.352941 8937 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:43.353133 8937 raft_consensus.cc:697] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [term 1 LEADER]: Becoming Leader. State: Replica: 267047fc785e4d8582c185fb64661b4b, State: Running, Role: LEADER
I20260501 02:09:43.353391 8937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } }
I20260501 02:09:43.353920 8932 sys_catalog.cc:565] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:43.354727 8938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "267047fc785e4d8582c185fb64661b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } } }
I20260501 02:09:43.354858 8938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:43.355216 8939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 267047fc785e4d8582c185fb64661b4b. Latest consensus state: current_term: 1 leader_uuid: "267047fc785e4d8582c185fb64661b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "267047fc785e4d8582c185fb64661b4b" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 39797 } } }
I20260501 02:09:43.355297 8939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:43.355640 8935 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:43.343308 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40993 (local address 127.18.96.254:39797)
0501 02:09:43.343676 (+ 368us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:43.343684 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:43.343713 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:43.344197 (+ 484us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:43.344211 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:43.344476 (+ 265us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:43.344711 (+ 235us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:43.345802 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:43.346789 (+ 987us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:43.347658 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:43.347881 (+ 223us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:43.350232 (+ 2351us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:43.350252 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:43.350273 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:43.350299 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:43.352169 (+ 1870us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:43.352657 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:43.352662 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:43.352684 (+ 22us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:43.352765 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:43.353222 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:43.353226 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:43.353228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:43.353576 (+ 348us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:43.354172 (+ 596us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:43.354728 (+ 556us) server_negotiation.cc:300] Negotiation successful
0501 02:09:43.354923 (+ 195us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
I20260501 02:09:43.358422 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
W20260501 02:09:43.358577 8952 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:43.358659 8952 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:43.358723 8945 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:43.359409 8945 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:43.367076 8945 catalog_manager.cc:1357] Generated new cluster ID: f892df3883184801afc78771365cc405
I20260501 02:09:43.367129 8945 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:43.368229 8935 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:43.360509 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51501 (local address 127.18.96.254:39797)
0501 02:09:43.360608 (+ 99us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:43.360627 (+ 19us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:43.361375 (+ 748us) server_negotiation.cc:408] Connection header received
0501 02:09:43.361478 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:43.361482 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:43.361526 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:43.361613 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:43.362506 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:43.363177 (+ 671us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:43.363754 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:43.363946 (+ 192us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:43.364910 (+ 964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:43.364926 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:43.364930 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:43.364956 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:43.366452 (+ 1496us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:43.367041 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:43.367045 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:43.367047 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:43.367091 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:43.367407 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:43.367411 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:43.367412 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:43.367589 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:43.367741 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:43.368015 (+ 274us) server_negotiation.cc:300] Negotiation successful
0501 02:09:43.368095 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260501 02:09:43.388471 8945 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:43.389521 8945 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:43.397769 8945 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 267047fc785e4d8582c185fb64661b4b: Generated new TSK 0
I20260501 02:09:43.398406 8945 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:43.403503 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8864
W20260501 02:09:43.409217 8846 connection.cc:570] client connection to 127.18.96.254:39797 recv error: Network error: failed to read from TLS socket (remote: 127.18.96.254:39797): Connection reset by peer (error 104)
2026-05-01T02:09:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8963](info): set up 2 sockets
May 01 02:09:43 dist-test-slave-2j90 krb5kdc[8963](info): commencing operation
krb5kdc: starting...
W20260501 02:09:45.366572 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:45 dist-test-slave-2j90 krb5kdc[8963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:45.494035 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:34543
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:34543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:45.589186 8979 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:45.589448 8979 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:45.589555 8979 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:45.592800 8979 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:45.592886 8979 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:45.592933 8979 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:45.593014 8979 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:45.593051 8979 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:45.597137 8979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:34543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:45.598141 8979 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:45.598902 8979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:45.604174 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:45.604271 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:45.604446 8979 server_base.cc:1061] running on GCE node
W20260501 02:09:45.604315 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:45.605082 8979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:45.605916 8979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:45.607134 8979 hybrid_clock.cc:648] HybridClock initialized: now 1777601385607111 us; error 34 us; skew 500 ppm
May 01 02:09:45 dist-test-slave-2j90 krb5kdc[8963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601385, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:45.609812 8979 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:45.610842 8979 webserver.cc:492] Webserver started at http://127.18.96.254:41137/ using document root <none> and password file <none>
I20260501 02:09:45.611379 8979 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:45.611455 8979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:45.611657 8979 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:45.613194 8979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "bcae2ecee19643af9aaa682cb05d4e4a"
format_stamp: "Formatted at 2026-05-01 02:09:45 on dist-test-slave-2j90"
server_key: "0b2283ca9525bfb64dea0d2e862062e6"
server_key_iv: "bc8e56e5ee08f3573c3181d7a7a2d99b"
server_key_version: "encryptionkey@0"
I20260501 02:09:45.613651 8979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "bcae2ecee19643af9aaa682cb05d4e4a"
format_stamp: "Formatted at 2026-05-01 02:09:45 on dist-test-slave-2j90"
server_key: "0b2283ca9525bfb64dea0d2e862062e6"
server_key_iv: "bc8e56e5ee08f3573c3181d7a7a2d99b"
server_key_version: "encryptionkey@0"
I20260501 02:09:45.616684 8979 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:45.618671 8994 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:45.619519 8979 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:45.619653 8979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "bcae2ecee19643af9aaa682cb05d4e4a"
format_stamp: "Formatted at 2026-05-01 02:09:45 on dist-test-slave-2j90"
server_key: "0b2283ca9525bfb64dea0d2e862062e6"
server_key_iv: "bc8e56e5ee08f3573c3181d7a7a2d99b"
server_key_version: "encryptionkey@0"
I20260501 02:09:45.619751 8979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:45.642360 8979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:45.644742 8979 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:45.644930 8979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:45.651815 8979 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:34543
I20260501 02:09:45.651839 9046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:34543 every 8 connection(s)
I20260501 02:09:45.652726 8979 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:45.655246 9047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:45.658972 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 8979
I20260501 02:09:45.659111 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:45.659338 18819 external_mini_cluster.cc:1468] Setting key 2108a9e0bf0f959c67c02704ac0a48cc
I20260501 02:09:45.660411 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a: Bootstrap starting.
I20260501 02:09:45.662510 9047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:45.663196 9047 log.cc:826] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:45.665016 9047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a: No bootstrap required, opened a new log
I20260501 02:09:45.666174 9050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:45.660350 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57419 (local address 127.18.96.254:34543)
0501 02:09:45.660724 (+ 374us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:45.660732 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:45.660761 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:45.661224 (+ 463us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:45.661237 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:45.661483 (+ 246us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:45.661707 (+ 224us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:45.662268 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:45.663027 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:45.663832 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:45.664086 (+ 254us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:45.664590 (+ 504us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:45.664606 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:45.664617 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:45.664929 (+ 312us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:45.664958 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:45.664972 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:45.665068 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:45.665435 (+ 367us) server_negotiation.cc:300] Negotiation successful
0501 02:09:45.665517 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260501 02:09:45.667764 9047 raft_consensus.cc:359] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } }
I20260501 02:09:45.667955 9047 raft_consensus.cc:385] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:45.668032 9047 raft_consensus.cc:740] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcae2ecee19643af9aaa682cb05d4e4a, State: Initialized, Role: FOLLOWER
I20260501 02:09:45.668388 9047 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } }
I20260501 02:09:45.668566 9047 raft_consensus.cc:399] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:45.668651 9047 raft_consensus.cc:493] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:45.668751 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:45.668771 9047 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:45.669699 9047 raft_consensus.cc:515] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } }
I20260501 02:09:45.670047 9047 leader_election.cc:304] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcae2ecee19643af9aaa682cb05d4e4a; no voters:
I20260501 02:09:45.670328 9047 leader_election.cc:290] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:45.670439 9056 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:45.671082 9056 raft_consensus.cc:697] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [term 1 LEADER]: Becoming Leader. State: Replica: bcae2ecee19643af9aaa682cb05d4e4a, State: Running, Role: LEADER
I20260501 02:09:45.671319 9047 sys_catalog.cc:565] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:45.671332 9056 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } }
I20260501 02:09:45.672821 9057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } } }
I20260501 02:09:45.672940 9057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:45.673298 9067 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:45.673295 9059 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcae2ecee19643af9aaa682cb05d4e4a. Latest consensus state: current_term: 1 leader_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcae2ecee19643af9aaa682cb05d4e4a" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 34543 } } }
I20260501 02:09:45.673465 9059 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:45.675485 9050 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:45.670862 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55507 (local address 127.18.96.254:34543)
0501 02:09:45.671004 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:45.671008 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:45.671585 (+ 577us) server_negotiation.cc:408] Connection header received
0501 02:09:45.671717 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:45.671720 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:45.671759 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:45.671848 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:45.672267 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:45.672810 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:45.673821 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:45.674024 (+ 203us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:45.674901 (+ 877us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:45.674916 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:45.674919 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:45.674930 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:45.674974 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:45.674986 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:45.675041 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:45.675323 (+ 282us) server_negotiation.cc:300] Negotiation successful
0501 02:09:45.675350 (+ 27us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260501 02:09:45.676308 9067 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:45.681249 9067 catalog_manager.cc:1357] Generated new cluster ID: 0e2d172780404f7dbb041aa964c7f572
I20260501 02:09:45.681313 9067 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:45.687206 9067 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:45.688194 9067 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:45.697675 9067 catalog_manager.cc:6044] T 00000000000000000000000000000000 P bcae2ecee19643af9aaa682cb05d4e4a: Generated new TSK 0
I20260501 02:09:45.698227 9067 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:45.702018 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 8979
W20260501 02:09:45.707856 8961 connection.cc:570] client connection to 127.18.96.254:34543 recv error: Network error: failed to read from TLS socket (remote: 127.18.96.254:34543): Connection reset by peer (error 104)
2026-05-01T02:09:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:45 dist-test-slave-2j90 krb5kdc[9078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:45 dist-test-slave-2j90 krb5kdc[9078](info): set up 2 sockets
May 01 02:09:45 dist-test-slave-2j90 krb5kdc[9078](info): commencing operation
krb5kdc: starting...
W20260501 02:09:47.674536 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:47 dist-test-slave-2j90 krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:47.809435 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46155
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:35373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:47.907426 9094 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:47.907636 9094 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:47.907722 9094 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:47.910926 9094 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:47.910975 9094 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:47.911024 9094 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:47.911057 9094 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:47.911114 9094 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:47.915252 9094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:35373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:46155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:47.916127 9094 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:47.916882 9094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:47.922309 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:47.922324 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:47.922369 9094 server_base.cc:1061] running on GCE node
W20260501 02:09:47.922305 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:47.923086 9094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:47.923906 9094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:47.925057 9094 hybrid_clock.cc:648] HybridClock initialized: now 1777601387925025 us; error 45 us; skew 500 ppm
May 01 02:09:47 dist-test-slave-2j90 krb5kdc[9078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601387, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:47.927677 9094 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:47.928614 9094 webserver.cc:492] Webserver started at http://127.18.96.254:38441/ using document root <none> and password file <none>
I20260501 02:09:47.929149 9094 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:47.929224 9094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:47.929430 9094 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:47.931061 9094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "e157902c188442719bf2f2a22e6dfdaa"
format_stamp: "Formatted at 2026-05-01 02:09:47 on dist-test-slave-2j90"
server_key: "e0b177ca75904b5bbd5d58ae7f316a41"
server_key_iv: "7875ca0d8c4fce68de2ea7735f858900"
server_key_version: "encryptionkey@0"
I20260501 02:09:47.931538 9094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "e157902c188442719bf2f2a22e6dfdaa"
format_stamp: "Formatted at 2026-05-01 02:09:47 on dist-test-slave-2j90"
server_key: "e0b177ca75904b5bbd5d58ae7f316a41"
server_key_iv: "7875ca0d8c4fce68de2ea7735f858900"
server_key_version: "encryptionkey@0"
I20260501 02:09:47.934691 9094 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:09:47.936690 9109 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:47.937597 9094 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260501 02:09:47.937732 9094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "e157902c188442719bf2f2a22e6dfdaa"
format_stamp: "Formatted at 2026-05-01 02:09:47 on dist-test-slave-2j90"
server_key: "e0b177ca75904b5bbd5d58ae7f316a41"
server_key_iv: "7875ca0d8c4fce68de2ea7735f858900"
server_key_version: "encryptionkey@0"
I20260501 02:09:47.937829 9094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:47.941932 9094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:47.948958 9094 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:47.949165 9094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:47.957069 9094 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:46155
I20260501 02:09:47.957077 9161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:46155 every 8 connection(s)
I20260501 02:09:47.958104 9094 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:47.960667 9162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:47.964138 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9094
I20260501 02:09:47.964339 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:47.964588 18819 external_mini_cluster.cc:1468] Setting key ca9b5de05fba617197777284551b406b
I20260501 02:09:47.966198 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa: Bootstrap starting.
I20260501 02:09:47.968556 9162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:47.969174 9165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:47.965687 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56775 (local address 127.18.96.254:46155)
0501 02:09:47.966070 (+ 383us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:47.966078 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:47.966108 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:09:47.966670 (+ 562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:47.966685 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:47.966901 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.967136 (+ 235us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:47.967951 (+ 815us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.967956 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:47.967965 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:47.967974 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:47.968003 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:47.968015 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.968096 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:47.968240 (+ 144us) server_negotiation.cc:300] Negotiation successful
0501 02:09:47.968369 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260501 02:09:47.969195 9162 log.cc:826] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:47.970995 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa: No bootstrap required, opened a new log
I20260501 02:09:47.971750 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:47.974037 9162 raft_consensus.cc:359] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } }
I20260501 02:09:47.974246 9162 raft_consensus.cc:385] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:47.974316 9162 raft_consensus.cc:740] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e157902c188442719bf2f2a22e6dfdaa, State: Initialized, Role: FOLLOWER
I20260501 02:09:47.974781 9162 consensus_queue.cc:260] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } }
I20260501 02:09:47.974948 9162 raft_consensus.cc:399] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:47.975008 9162 raft_consensus.cc:493] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:47.975092 9162 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:47.975903 9165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:47.973906 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36937 (local address 127.18.96.254:46155)
0501 02:09:47.974061 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:47.974065 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:47.974288 (+ 223us) server_negotiation.cc:408] Connection header received
0501 02:09:47.974455 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:47.974458 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:47.974487 (+ 29us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:47.974602 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:47.975477 (+ 875us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:47.975481 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:47.975484 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:47.975496 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:47.975533 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:47.975545 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:47.975651 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:47.975742 (+ 91us) server_negotiation.cc:300] Negotiation successful
0501 02:09:47.975775 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260501 02:09:47.975921 9162 raft_consensus.cc:515] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } }
I20260501 02:09:47.976351 9162 leader_election.cc:304] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e157902c188442719bf2f2a22e6dfdaa; no voters:
I20260501 02:09:47.976675 9162 leader_election.cc:290] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:47.976809 9172 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:47.977051 9172 raft_consensus.cc:697] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [term 1 LEADER]: Becoming Leader. State: Replica: e157902c188442719bf2f2a22e6dfdaa, State: Running, Role: LEADER
I20260501 02:09:47.977352 9172 consensus_queue.cc:237] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } }
I20260501 02:09:47.977718 9162 sys_catalog.cc:565] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:47.979030 9174 sys_catalog.cc:455] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e157902c188442719bf2f2a22e6dfdaa. Latest consensus state: current_term: 1 leader_uuid: "e157902c188442719bf2f2a22e6dfdaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } } }
I20260501 02:09:47.979144 9174 sys_catalog.cc:458] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:47.979491 9181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:47.980255 9173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e157902c188442719bf2f2a22e6dfdaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e157902c188442719bf2f2a22e6dfdaa" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 46155 } } }
I20260501 02:09:47.980355 9173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:47.982534 9181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:47.987107 9181 catalog_manager.cc:1357] Generated new cluster ID: 039efde8f3cd456bbcfa0520f4a86556
I20260501 02:09:47.987169 9181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:48.012619 9181 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:48.012935 9181 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:48.018724 9181 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e157902c188442719bf2f2a22e6dfdaa: Generated new TSK 0
I20260501 02:09:48.019306 9181 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:48.026640 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9094
2026-05-01T02:09:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:48 dist-test-slave-2j90 krb5kdc[9193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:48 dist-test-slave-2j90 krb5kdc[9193](info): set up 2 sockets
May 01 02:09:48 dist-test-slave-2j90 krb5kdc[9193](info): commencing operation
krb5kdc: starting...
W20260501 02:09:50.010397 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.964s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:50.144930 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38301
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:50.241278 9209 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:50.241520 9209 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:50.241597 9209 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:50.244853 9209 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:50.244939 9209 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:50.244984 9209 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:50.245046 9209 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:50.245090 9209 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:50.249276 9209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:50.250293 9209 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:50.251096 9209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:50.256335 9217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:50.256325 9214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:50.256325 9215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:50.256711 9209 server_base.cc:1061] running on GCE node
I20260501 02:09:50.257174 9209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:50.258111 9209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:50.259284 9209 hybrid_clock.cc:648] HybridClock initialized: now 1777601390259267 us; error 37 us; skew 500 ppm
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601390, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:50.261775 9209 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:50.262816 9209 webserver.cc:492] Webserver started at http://127.18.96.254:39367/ using document root <none> and password file <none>
I20260501 02:09:50.263317 9209 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:50.263394 9209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:50.263599 9209 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:50.265174 9209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "d29d80389bf445a899e3141990eddc18"
format_stamp: "Formatted at 2026-05-01 02:09:50 on dist-test-slave-2j90"
server_key: "f638d240fcf62bcd66e4b8d56c727ea2"
server_key_iv: "aedf887a3392333fbe76b6ac0469222e"
server_key_version: "encryptionkey@0"
I20260501 02:09:50.265654 9209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "d29d80389bf445a899e3141990eddc18"
format_stamp: "Formatted at 2026-05-01 02:09:50 on dist-test-slave-2j90"
server_key: "f638d240fcf62bcd66e4b8d56c727ea2"
server_key_iv: "aedf887a3392333fbe76b6ac0469222e"
server_key_version: "encryptionkey@0"
I20260501 02:09:50.268879 9209 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:50.270915 9224 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:50.271823 9209 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260501 02:09:50.271958 9209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "d29d80389bf445a899e3141990eddc18"
format_stamp: "Formatted at 2026-05-01 02:09:50 on dist-test-slave-2j90"
server_key: "f638d240fcf62bcd66e4b8d56c727ea2"
server_key_iv: "aedf887a3392333fbe76b6ac0469222e"
server_key_version: "encryptionkey@0"
I20260501 02:09:50.272053 9209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:50.289206 9209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:50.291510 9209 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:50.291699 9209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:50.298182 9209 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38301
I20260501 02:09:50.298203 9276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38301 every 8 connection(s)
I20260501 02:09:50.299084 9209 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:50.299829 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9209
I20260501 02:09:50.299952 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:50.300212 18819 external_mini_cluster.cc:1468] Setting key dc12f86ad6dc01e74cce92ff46585488
I20260501 02:09:50.302068 9277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:50.308043 9277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18: Bootstrap starting.
I20260501 02:09:50.310524 9277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:50.311311 9277 log.cc:826] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:50.312368 9280 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:50.301298 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57675 (local address 127.18.96.254:38301)
0501 02:09:50.301640 (+ 342us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:50.301648 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:50.301678 (+ 30us) server_negotiation.cc:408] Connection header received
0501 02:09:50.302364 (+ 686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:50.302400 (+ 36us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:50.302658 (+ 258us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:50.302898 (+ 240us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:50.303957 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:50.304722 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:50.305597 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:50.305804 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:50.308208 (+ 2404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:50.308264 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:50.308276 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:50.308305 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:50.310146 (+ 1841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:50.310532 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:50.310540 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:50.310546 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:50.310623 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:50.310823 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:50.310825 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:50.310827 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:50.311175 (+ 348us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:50.311352 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:50.311587 (+ 235us) server_negotiation.cc:300] Negotiation successful
0501 02:09:50.311803 (+ 216us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260501 02:09:50.313094 9277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18: No bootstrap required, opened a new log
I20260501 02:09:50.314875 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:50.315703 9277 raft_consensus.cc:359] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } }
I20260501 02:09:50.315882 9277 raft_consensus.cc:385] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:50.316002 9277 raft_consensus.cc:740] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29d80389bf445a899e3141990eddc18, State: Initialized, Role: FOLLOWER
I20260501 02:09:50.316397 9277 consensus_queue.cc:260] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } }
I20260501 02:09:50.316529 9277 raft_consensus.cc:399] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:50.316593 9277 raft_consensus.cc:493] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:50.316679 9277 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:50.317620 9277 raft_consensus.cc:515] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } }
I20260501 02:09:50.317951 9277 leader_election.cc:304] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d29d80389bf445a899e3141990eddc18; no voters:
I20260501 02:09:50.318240 9277 leader_election.cc:290] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:50.318704 9287 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:50.318922 9287 raft_consensus.cc:697] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [term 1 LEADER]: Becoming Leader. State: Replica: d29d80389bf445a899e3141990eddc18, State: Running, Role: LEADER
I20260501 02:09:50.319216 9287 consensus_queue.cc:237] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } }
I20260501 02:09:50.319272 9277 sys_catalog.cc:565] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:50.321017 9289 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d29d80389bf445a899e3141990eddc18. Latest consensus state: current_term: 1 leader_uuid: "d29d80389bf445a899e3141990eddc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } } }
I20260501 02:09:50.321014 9288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d29d80389bf445a899e3141990eddc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d29d80389bf445a899e3141990eddc18" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38301 } } }
I20260501 02:09:50.321154 9289 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:50.321170 9288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:50.321849 9296 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:50.324587 9280 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:50.316721 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33817 (local address 127.18.96.254:38301)
0501 02:09:50.316845 (+ 124us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:50.316851 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:50.317025 (+ 174us) server_negotiation.cc:408] Connection header received
0501 02:09:50.317089 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:50.317092 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:50.317135 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:50.317212 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:50.317931 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:50.318614 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:50.319393 (+ 779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:50.319576 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:50.321041 (+ 1465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:50.321060 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:50.321063 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:50.321092 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:50.322821 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:50.323266 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:50.323270 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:50.323271 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:50.323333 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:50.323582 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:50.323584 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:50.323586 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:50.323905 (+ 319us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:50.324072 (+ 167us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:50.324314 (+ 242us) server_negotiation.cc:300] Negotiation successful
0501 02:09:50.324423 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:09:50.325296 9296 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:50.330047 9296 catalog_manager.cc:1357] Generated new cluster ID: a2a3841eccac4558bca9e9884d735aa9
I20260501 02:09:50.330103 9296 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:50.353013 9296 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:50.353850 9296 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:50.363679 9296 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d29d80389bf445a899e3141990eddc18: Generated new TSK 0
I20260501 02:09:50.364243 9296 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:50.368603 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9209
2026-05-01T02:09:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9308](info): set up 2 sockets
May 01 02:09:50 dist-test-slave-2j90 krb5kdc[9308](info): commencing operation
krb5kdc: starting...
W20260501 02:09:52.350612 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.962s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:52 dist-test-slave-2j90 krb5kdc[9308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:52.481562 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38803
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:52.578524 9324 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:52.578794 9324 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:52.578886 9324 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:52.582011 9324 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:52.582099 9324 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:52.582142 9324 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:52.582201 9324 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:52.582242 9324 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:52.586349 9324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:52.587316 9324 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:52.588076 9324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:52.593173 9330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:52.593168 9329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:52.593279 9324 server_base.cc:1061] running on GCE node
W20260501 02:09:52.593168 9332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:52.593761 9324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:52.594578 9324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:52.595779 9324 hybrid_clock.cc:648] HybridClock initialized: now 1777601392595766 us; error 45 us; skew 500 ppm
May 01 02:09:52 dist-test-slave-2j90 krb5kdc[9308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601392, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:52.598305 9324 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:52.599130 9324 webserver.cc:492] Webserver started at http://127.18.96.254:38141/ using document root <none> and password file <none>
I20260501 02:09:52.599573 9324 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:52.599615 9324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:52.599750 9324 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:52.601153 9324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "c97ade7043774192b46dc704bb7ef717"
format_stamp: "Formatted at 2026-05-01 02:09:52 on dist-test-slave-2j90"
server_key: "ee4155da659c5e118e2d98eda8a889cb"
server_key_iv: "b454b74256b3df91af4debdbd5e0459d"
server_key_version: "encryptionkey@0"
I20260501 02:09:52.601586 9324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "c97ade7043774192b46dc704bb7ef717"
format_stamp: "Formatted at 2026-05-01 02:09:52 on dist-test-slave-2j90"
server_key: "ee4155da659c5e118e2d98eda8a889cb"
server_key_iv: "b454b74256b3df91af4debdbd5e0459d"
server_key_version: "encryptionkey@0"
I20260501 02:09:52.604591 9324 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260501 02:09:52.606475 9339 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:52.607348 9324 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:52.607458 9324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "c97ade7043774192b46dc704bb7ef717"
format_stamp: "Formatted at 2026-05-01 02:09:52 on dist-test-slave-2j90"
server_key: "ee4155da659c5e118e2d98eda8a889cb"
server_key_iv: "b454b74256b3df91af4debdbd5e0459d"
server_key_version: "encryptionkey@0"
I20260501 02:09:52.607528 9324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:52.615365 9324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:52.617527 9324 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:52.617653 9324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:52.624154 9324 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38803
I20260501 02:09:52.624152 9391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38803 every 8 connection(s)
I20260501 02:09:52.625061 9324 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:52.626438 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9324
I20260501 02:09:52.626591 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:52.626875 18819 external_mini_cluster.cc:1468] Setting key c46b7ff04fb6743ba407b2c78282a3e1
I20260501 02:09:52.627935 9392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:52.632966 9392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717: Bootstrap starting.
I20260501 02:09:52.633630 9395 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:52.627962 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39473 (local address 127.18.96.254:38803)
0501 02:09:52.628327 (+ 365us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:52.628336 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:52.628364 (+ 28us) server_negotiation.cc:408] Connection header received
0501 02:09:52.628948 (+ 584us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:52.628963 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:52.629212 (+ 249us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:52.629517 (+ 305us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:52.630074 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:52.630829 (+ 755us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:52.631534 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:52.631809 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:52.632272 (+ 463us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:52.632335 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:52.632344 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:52.632695 (+ 351us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:52.632725 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:52.632740 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:52.632851 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:52.633016 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:09:52.633136 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
I20260501 02:09:52.635054 9392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:52.635820 9392 log.cc:826] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:52.636075 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:52.637705 9392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717: No bootstrap required, opened a new log
I20260501 02:09:52.640511 9392 raft_consensus.cc:359] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } }
I20260501 02:09:52.640700 9392 raft_consensus.cc:385] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:52.640765 9392 raft_consensus.cc:740] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97ade7043774192b46dc704bb7ef717, State: Initialized, Role: FOLLOWER
I20260501 02:09:52.641220 9395 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:52.638083 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39039 (local address 127.18.96.254:38803)
0501 02:09:52.638198 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:52.638202 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:52.638371 (+ 169us) server_negotiation.cc:408] Connection header received
0501 02:09:52.638455 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:52.638458 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:52.638502 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:52.638578 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:52.638899 (+ 321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:52.639560 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:52.640101 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:52.640302 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:52.640730 (+ 428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:52.640744 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:52.640747 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:52.640765 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:52.640787 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:52.640796 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:52.640875 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:52.641079 (+ 204us) server_negotiation.cc:300] Negotiation successful
0501 02:09:52.641133 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260501 02:09:52.641232 9392 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } }
I20260501 02:09:52.641448 9392 raft_consensus.cc:399] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:52.641511 9392 raft_consensus.cc:493] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:52.641597 9392 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:52.642359 9392 raft_consensus.cc:515] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } }
I20260501 02:09:52.642660 9392 leader_election.cc:304] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97ade7043774192b46dc704bb7ef717; no voters:
I20260501 02:09:52.642906 9392 leader_election.cc:290] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:52.643019 9402 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:52.643218 9402 raft_consensus.cc:697] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [term 1 LEADER]: Becoming Leader. State: Replica: c97ade7043774192b46dc704bb7ef717, State: Running, Role: LEADER
I20260501 02:09:52.643517 9402 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } }
I20260501 02:09:52.643870 9392 sys_catalog.cc:565] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:52.645058 9404 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97ade7043774192b46dc704bb7ef717. Latest consensus state: current_term: 1 leader_uuid: "c97ade7043774192b46dc704bb7ef717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } } }
I20260501 02:09:52.645167 9404 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:52.645540 9407 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:52.645503 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97ade7043774192b46dc704bb7ef717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97ade7043774192b46dc704bb7ef717" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38803 } } }
I20260501 02:09:52.645704 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:52.647974 9407 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:52.652820 9407 catalog_manager.cc:1357] Generated new cluster ID: 2c936fdc7f974b77ab6df6b530ed1a1d
I20260501 02:09:52.652891 9407 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:52.666993 9407 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:52.667989 9407 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:52.681989 9407 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c97ade7043774192b46dc704bb7ef717: Generated new TSK 0
I20260501 02:09:52.682610 9407 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:52.690577 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9324
2026-05-01T02:09:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:52 dist-test-slave-2j90 krb5kdc[9423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:52 dist-test-slave-2j90 krb5kdc[9423](info): set up 2 sockets
May 01 02:09:52 dist-test-slave-2j90 krb5kdc[9423](info): commencing operation
krb5kdc: starting...
W20260501 02:09:54.666575 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.956s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:54 dist-test-slave-2j90 krb5kdc[9423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:54.797107 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43911
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:43911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:54.895902 9439 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:54.896168 9439 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:54.896262 9439 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:54.899510 9439 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:54.899611 9439 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:54.899669 9439 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:54.899734 9439 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:54.899773 9439 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:54.903918 9439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:43911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:54.904937 9439 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:54.905774 9439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:54.911216 9445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:54.911185 9444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:54.911211 9447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:54.911319 9439 server_base.cc:1061] running on GCE node
I20260501 02:09:54.911842 9439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:54.912698 9439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:54.913852 9439 hybrid_clock.cc:648] HybridClock initialized: now 1777601394913827 us; error 36 us; skew 500 ppm
May 01 02:09:54 dist-test-slave-2j90 krb5kdc[9423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601394, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:54.916385 9439 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:54.917387 9439 webserver.cc:492] Webserver started at http://127.18.96.254:45287/ using document root <none> and password file <none>
I20260501 02:09:54.917907 9439 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:54.918005 9439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:54.918224 9439 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:54.919807 9439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c"
format_stamp: "Formatted at 2026-05-01 02:09:54 on dist-test-slave-2j90"
server_key: "537d3a42be73ab511faa705ce5161b64"
server_key_iv: "c894e23a95eeb6ecb2bde01ebb116412"
server_key_version: "encryptionkey@0"
I20260501 02:09:54.920264 9439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c"
format_stamp: "Formatted at 2026-05-01 02:09:54 on dist-test-slave-2j90"
server_key: "537d3a42be73ab511faa705ce5161b64"
server_key_iv: "c894e23a95eeb6ecb2bde01ebb116412"
server_key_version: "encryptionkey@0"
I20260501 02:09:54.923338 9439 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:54.925293 9454 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:54.926297 9439 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:54.926433 9439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c"
format_stamp: "Formatted at 2026-05-01 02:09:54 on dist-test-slave-2j90"
server_key: "537d3a42be73ab511faa705ce5161b64"
server_key_iv: "c894e23a95eeb6ecb2bde01ebb116412"
server_key_version: "encryptionkey@0"
I20260501 02:09:54.926541 9439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:54.931581 9439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:54.938702 9439 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:54.938891 9439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:54.945639 9439 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:43911
I20260501 02:09:54.945653 9506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:43911 every 8 connection(s)
I20260501 02:09:54.946833 9439 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:54.949385 9507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:54.952088 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9439
I20260501 02:09:54.952332 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:54.952567 18819 external_mini_cluster.cc:1468] Setting key 795710689459817b35805a76cf3c314e
I20260501 02:09:54.955101 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c: Bootstrap starting.
I20260501 02:09:54.957070 9510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:54.953656 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40511 (local address 127.18.96.254:43911)
0501 02:09:54.954057 (+ 401us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:54.954068 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:54.954105 (+ 37us) server_negotiation.cc:408] Connection header received
0501 02:09:54.954707 (+ 602us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:54.954729 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:54.955016 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:54.955371 (+ 355us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:54.956035 (+ 664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:54.956041 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:54.956053 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:54.956064 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:54.956100 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:54.956120 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:54.956263 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:54.956351 (+ 88us) server_negotiation.cc:300] Negotiation successful
0501 02:09:54.956451 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
I20260501 02:09:54.957520 9507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:54.958308 9507 log.cc:826] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:54.959661 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:54.960153 9507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c: No bootstrap required, opened a new log
I20260501 02:09:54.963450 9510 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:54.961849 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55447 (local address 127.18.96.254:43911)
0501 02:09:54.962001 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:54.962006 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:54.962177 (+ 171us) server_negotiation.cc:408] Connection header received
0501 02:09:54.962281 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:54.962283 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:54.962311 (+ 28us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:54.962386 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:54.963045 (+ 659us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:54.963048 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:54.963050 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0501 02:09:54.963058 (+ 8us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:54.963081 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0501 02:09:54.963091 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:54.963156 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:54.963267 (+ 111us) server_negotiation.cc:300] Negotiation successful
0501 02:09:54.963291 (+ 24us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:09:54.963418 9507 raft_consensus.cc:359] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } }
I20260501 02:09:54.963619 9507 raft_consensus.cc:385] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260501 02:09:54.963685 9507 raft_consensus.cc:740] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ed1a72ee7944a1b8916ebfa2bbed5c, State: Initialized, Role: FOLLOWER
I20260501 02:09:54.964066 9507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } }
I20260501 02:09:54.964178 9507 raft_consensus.cc:399] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260501 02:09:54.964268 9507 raft_consensus.cc:493] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260501 02:09:54.964382 9507 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:54.965191 9507 raft_consensus.cc:515] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } }
I20260501 02:09:54.965451 9507 leader_election.cc:304] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ed1a72ee7944a1b8916ebfa2bbed5c; no voters:
I20260501 02:09:54.965672 9507 leader_election.cc:290] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260501 02:09:54.965795 9517 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:54.966028 9517 raft_consensus.cc:697] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [term 1 LEADER]: Becoming Leader. State: Replica: 39ed1a72ee7944a1b8916ebfa2bbed5c, State: Running, Role: LEADER
I20260501 02:09:54.966290 9517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } }
I20260501 02:09:54.966874 9507 sys_catalog.cc:565] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:54.967980 9519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ed1a72ee7944a1b8916ebfa2bbed5c. Latest consensus state: current_term: 1 leader_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } } }
I20260501 02:09:54.968088 9518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ed1a72ee7944a1b8916ebfa2bbed5c" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 43911 } } }
I20260501 02:09:54.968122 9519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:54.968149 9518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:54.968464 9526 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:54.971294 9526 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:54.976279 9526 catalog_manager.cc:1357] Generated new cluster ID: c214cb1d1f9d433f9460ea4d8c0edef7
I20260501 02:09:54.976356 9526 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:54.985498 9526 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:54.985769 9526 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:54.993353 9526 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 39ed1a72ee7944a1b8916ebfa2bbed5c: Generated new TSK 0
I20260501 02:09:54.993827 9526 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260501 02:09:55.003311 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9439
2026-05-01T02:09:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2313 ms)
[----------] 18 tests from AuthTokenIssuingTest (41899 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:55 dist-test-slave-2j90 krb5kdc[9538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:55 dist-test-slave-2j90 krb5kdc[9538](info): set up 2 sockets
May 01 02:09:55 dist-test-slave-2j90 krb5kdc[9538](info): commencing operation
krb5kdc: starting...
W20260501 02:09:56.970644 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.947s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:09:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:09:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:57.101897 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38455
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:09:57.203478 9554 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260501 02:09:57.203743 9554 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:57.203835 9554 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:57.203962 9554 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:57.207237 9554 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:57.207314 9554 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:57.207376 9554 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:57.207417 9554 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:57.207473 9554 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:57.211727 9554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:57.212708 9554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:57.213477 9554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:57.218822 9562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.218788 9559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.218783 9560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:57.219133 9554 server_base.cc:1061] running on GCE node
I20260501 02:09:57.219569 9554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:57.220363 9554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:57.221515 9554 hybrid_clock.cc:648] HybridClock initialized: now 1777601397221492 us; error 33 us; skew 500 ppm
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:57.224032 9554 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:09:57.224931 9554 webserver.cc:492] Webserver started at http://127.18.96.254:44765/ using document root <none> and password file <none>
I20260501 02:09:57.225378 9554 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:57.225430 9554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:57.225570 9554 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:57.227123 9554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "e49648785c78474b9864aa728c098774"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "d87f286a879ac50f47201f553ecfd696"
server_key_iv: "44c6f2195425475db93e1b2b9d5e9501"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.227532 9554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "e49648785c78474b9864aa728c098774"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "d87f286a879ac50f47201f553ecfd696"
server_key_iv: "44c6f2195425475db93e1b2b9d5e9501"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.230376 9554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260501 02:09:57.232263 9569 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:57.233139 9554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:57.233223 9554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "e49648785c78474b9864aa728c098774"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "d87f286a879ac50f47201f553ecfd696"
server_key_iv: "44c6f2195425475db93e1b2b9d5e9501"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.233496 9554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:57.244246 9554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:57.246552 9554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:57.246701 9554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:57.253369 9554 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:38455
I20260501 02:09:57.253396 9621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:38455 every 8 connection(s)
I20260501 02:09:57.254285 9554 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:09:57.256783 9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:57.256847 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9554
I20260501 02:09:57.257004 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:09:57.257256 18819 external_mini_cluster.cc:1468] Setting key f2550240adb0ef256d0a357f14e5fcbc
I20260501 02:09:57.260867 9622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.253@KRBTEST.COM" created.
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:57.273274 9626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.262633 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.18.96.254:38455)
0501 02:09:57.263094 (+ 461us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.263103 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.263241 (+ 138us) server_negotiation.cc:408] Connection header received
0501 02:09:57.264294 (+ 1053us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.264298 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.264487 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.264613 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.265490 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.266328 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.267050 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.267223 (+ 173us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.269192 (+ 1969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.269208 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.269216 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.269236 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.270852 (+ 1616us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.271477 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.271481 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.271482 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.271533 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.271851 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.271853 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.271854 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.272091 (+ 237us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.272184 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.272595 (+ 411us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.272742 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"spinlock_wait_cycles":46336,"thread_start_us":133,"threads_started":1}
I20260501 02:09:57.273285 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.262633 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38455 (local address 127.0.0.1:52088)
0501 02:09:57.263098 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.263133 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.263997 (+ 864us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.264671 (+ 674us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.264679 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.264989 (+ 310us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.265325 (+ 336us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.265338 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.266446 (+ 1108us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.266449 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.266916 (+ 467us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.266924 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.267129 (+ 205us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.267736 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.267752 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.269047 (+ 1295us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.270984 (+ 1937us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.270996 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.271005 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.271320 (+ 315us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.271639 (+ 319us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.271644 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.271647 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.271758 (+ 111us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.272206 (+ 448us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.272215 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.272351 (+ 136us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.272553 (+ 202us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.272861 (+ 308us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":92,"thread_start_us":119,"threads_started":1}
I20260501 02:09:57.275508 9622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.276190 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.275942 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35214)
0501 02:09:57.276049 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.276102 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260501 02:09:57.276374 9570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
I20260501 02:09:57.277170 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.276987 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35220)
0501 02:09:57.277092 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.277127 (+ 35us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260501 02:09:57.277397 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41649: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:57.301110 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41649
--webserver_interface=127.18.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:09:57.326606 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } attempt: 1
I20260501 02:09:57.327428 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.327131 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35222)
0501 02:09:57.327288 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.327349 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260501 02:09:57.328132 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.327929 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35230)
0501 02:09:57.328039 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.328082 (+ 43us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260501 02:09:57.328261 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41649: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
I20260501 02:09:57.396433 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } attempt: 2
I20260501 02:09:57.397217 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.396922 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35234)
0501 02:09:57.397079 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.397140 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260501 02:09:57.397749 9631 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260501 02:09:57.397930 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.397724 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35246)
0501 02:09:57.397834 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.397879 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260501 02:09:57.398026 9631 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:57.398087 9631 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:57.398164 9631 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:57.398149 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41649: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41649: connect: Connection refused (error 111)
W20260501 02:09:57.401387 9631 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:57.401436 9631 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:57.401459 9631 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:57.401530 9631 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:57.401568 9631 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:57.405759 9631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:57.406760 9631 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:57.407554 9631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:57.413195 9636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.413218 9637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.413187 9639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:57.413666 9631 server_base.cc:1061] running on GCE node
I20260501 02:09:57.414153 9631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:57.414980 9631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:57.416184 9631 hybrid_clock.cc:648] HybridClock initialized: now 1777601397416157 us; error 35 us; skew 500 ppm
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:57.418821 9631 init.cc:377] Logged in from keytab as kudu/127.18.96.253@KRBTEST.COM (short username kudu)
I20260501 02:09:57.419759 9631 webserver.cc:492] Webserver started at http://127.18.96.253:41589/ using document root <none> and password file <none>
I20260501 02:09:57.420270 9631 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:57.420341 9631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:57.420512 9631 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:57.422077 9631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data/instance:
uuid: "5cfc1426daf5406f8110b8e077366fdf"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "e7a6bd3f9760b03a3bebb9b301cfe407"
server_key_iv: "35f22c2cf4bf1e6c2ab31a682ed62247"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.422531 9631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal/instance:
uuid: "5cfc1426daf5406f8110b8e077366fdf"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "e7a6bd3f9760b03a3bebb9b301cfe407"
server_key_iv: "35f22c2cf4bf1e6c2ab31a682ed62247"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.425659 9631 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260501 02:09:57.427659 9646 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:57.428670 9631 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:57.428813 9631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "5cfc1426daf5406f8110b8e077366fdf"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "e7a6bd3f9760b03a3bebb9b301cfe407"
server_key_iv: "35f22c2cf4bf1e6c2ab31a682ed62247"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.428918 9631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:57.469206 9631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:57.471848 9631 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:57.472047 9631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:57.479578 9698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.253:41649 every 8 connection(s)
I20260501 02:09:57.479583 9631 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.253:41649
I20260501 02:09:57.480611 9631 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
I20260501 02:09:57.483222 9699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:57.486398 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9631
I20260501 02:09:57.486549 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-1/wal/instance
I20260501 02:09:57.486788 18819 external_mini_cluster.cc:1468] Setting key cd8c9715bd4a9a1011c193992be5ce2d
I20260501 02:09:57.486826 9699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.489312 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } attempt: 3
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.96.252@KRBTEST.COM; defaulting to no policy
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.252@KRBTEST.COM" created.
I20260501 02:09:57.498999 9626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.488419 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.18.96.254:38455)
0501 02:09:57.488582 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.488587 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.489174 (+ 587us) server_negotiation.cc:408] Connection header received
0501 02:09:57.490131 (+ 957us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.490135 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.490179 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.490227 (+ 48us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.491453 (+ 1226us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.491927 (+ 474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.492531 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.492722 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.494990 (+ 2268us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.495010 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.495013 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.495040 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.496565 (+ 1525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.497223 (+ 658us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.497226 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.497227 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.497268 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.497629 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.497645 (+ 16us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.497647 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.497848 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.497958 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.498624 (+ 666us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.498748 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":1408}
I20260501 02:09:57.499403 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.489716 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35254)
0501 02:09:57.489897 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.489911 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.490040 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.490802 (+ 762us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.490809 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.491020 (+ 211us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.491319 (+ 299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.491329 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.492770 (+ 1441us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.492775 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.493330 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.493340 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.493457 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.493938 (+ 481us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.493959 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.495413 (+ 1454us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.497914 (+ 2501us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.497920 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.497924 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.498151 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.498479 (+ 328us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.498485 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.498490 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.498576 (+ 86us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.499001 (+ 425us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.499005 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.499079 (+ 74us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.499153 (+ 74us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.499257 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"spinlock_wait_cycles":6016}
I20260501 02:09:57.499466 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.488652 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38455 (local address 127.0.0.1:52100)
0501 02:09:57.489035 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.489072 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.489879 (+ 807us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.490478 (+ 599us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.490486 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.490974 (+ 488us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.491303 (+ 329us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.491314 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.492042 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.492047 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.492415 (+ 368us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.492421 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.492614 (+ 193us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.493221 (+ 607us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.493238 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.494811 (+ 1573us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.496696 (+ 1885us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.496703 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.496708 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.497121 (+ 413us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.497362 (+ 241us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.497365 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.497368 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.497491 (+ 123us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.498034 (+ 543us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.498040 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.498287 (+ 247us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.498526 (+ 239us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.498770 (+ 244us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":115,"threads_started":1}
I20260501 02:09:57.500205 9703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.489824 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.18.96.253:41649)
0501 02:09:57.490268 (+ 444us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.490273 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.490305 (+ 32us) server_negotiation.cc:408] Connection header received
0501 02:09:57.490448 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.490453 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.490608 (+ 155us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.490757 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.491521 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.492632 (+ 1111us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.493714 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.494660 (+ 946us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.495755 (+ 1095us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.495770 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.495777 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.495798 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.497813 (+ 2015us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.498278 (+ 465us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.498286 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.498287 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.498359 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.498689 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.498692 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.498694 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.498885 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.499573 (+ 688us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.499927 (+ 354us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.500039 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":178,"threads_started":1}
I20260501 02:09:57.500645 9699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.502027 9622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.502620 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.502420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44806)
0501 02:09:57.502528 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.502574 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260501 02:09:57.503284 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.503077 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44820)
0501 02:09:57.503194 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.503232 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260501 02:09:57.503428 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:09:57.508507 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.501172 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35264)
0501 02:09:57.501357 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.501371 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.501460 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.502020 (+ 560us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.502025 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.502200 (+ 175us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.502440 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.502449 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.503436 (+ 987us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.503441 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.503963 (+ 522us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.503974 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.504088 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.504508 (+ 420us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.504527 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.505714 (+ 1187us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.507328 (+ 1614us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.507330 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.507332 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.507494 (+ 162us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.507725 (+ 231us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.507730 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.507735 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.507824 (+ 89us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.508107 (+ 283us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.508109 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.508172 (+ 63us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.508251 (+ 79us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.508327 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260501 02:09:57.508582 9703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.501227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.18.96.253:41649)
0501 02:09:57.501339 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.501344 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.501438 (+ 94us) server_negotiation.cc:408] Connection header received
0501 02:09:57.501550 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.501553 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.501594 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.501676 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.502576 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.503316 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.504080 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.504223 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.505841 (+ 1618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.505857 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.505859 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.505901 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.507197 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.507578 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.507580 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.507581 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.507616 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.507909 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.507911 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.507912 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.508017 (+ 105us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.508087 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.508313 (+ 226us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.508393 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260501 02:09:57.509399 9699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.510120 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.509774 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44822)
0501 02:09:57.509940 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.510032 (+ 92us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260501 02:09:57.510349 9648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
I20260501 02:09:57.511119 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.510948 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44838)
0501 02:09:57.511039 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.511076 (+ 37us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260501 02:09:57.511269 9699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:09:57.534444 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:44467
--webserver_interface=127.18.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:36157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:09:57.534581 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 1
I20260501 02:09:57.535316 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.535040 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44846)
0501 02:09:57.535192 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.535246 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260501 02:09:57.536011 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.535807 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44858)
0501 02:09:57.535915 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.535953 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260501 02:09:57.536190 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
I20260501 02:09:57.560494 9699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 1
I20260501 02:09:57.561367 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.561048 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44866)
0501 02:09:57.561220 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.561279 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260501 02:09:57.562265 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.562038 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44868)
0501 02:09:57.562153 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.562199 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260501 02:09:57.562408 9699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
I20260501 02:09:57.611431 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 2
I20260501 02:09:57.612316 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.611992 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44874)
0501 02:09:57.612185 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.612240 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260501 02:09:57.613104 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.612887 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44886)
0501 02:09:57.613005 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.613051 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260501 02:09:57.613234 9622 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
I20260501 02:09:57.630560 9699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 2
I20260501 02:09:57.631263 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.630973 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44900)
0501 02:09:57.631136 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.631187 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260501 02:09:57.631985 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.631785 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44916)
0501 02:09:57.631891 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.631931 (+ 40us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260501 02:09:57.632192 9699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:44467: Network error: Client connection negotiation failed: client connection to 127.18.96.252:44467: connect: Connection refused (error 111)
W20260501 02:09:57.632792 9708 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260501 02:09:57.633023 9708 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:09:57.633067 9708 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:09:57.633157 9708 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:09:57.636430 9708 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:09:57.636478 9708 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:09:57.636502 9708 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:09:57.636519 9708 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:09:57.636580 9708 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:09:57.640787 9708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:36157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:38455,127.18.96.253:41649,127.18.96.252:44467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:44467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:09:57.641656 9708 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:09:57.642448 9708 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:09:57.647353 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.647365 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:09:57.647384 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:09:57.647644 9708 server_base.cc:1061] running on GCE node
I20260501 02:09:57.648047 9708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:09:57.648828 9708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:09:57.650028 9708 hybrid_clock.cc:648] HybridClock initialized: now 1777601397650007 us; error 36 us; skew 500 ppm
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:09:57.652652 9708 init.cc:377] Logged in from keytab as kudu/127.18.96.252@KRBTEST.COM (short username kudu)
I20260501 02:09:57.653537 9708 webserver.cc:492] Webserver started at http://127.18.96.252:38435/ using document root <none> and password file <none>
I20260501 02:09:57.654107 9708 fs_manager.cc:362] Metadata directory not provided
I20260501 02:09:57.654191 9708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:09:57.654367 9708 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:09:57.655853 9708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data/instance:
uuid: "3d4db9ea78bd4a0e9e34b805f7932e48"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "66ede35f79ba337355a8a79b5e19cb36"
server_key_iv: "ee4fe21bb0a9245adde97c1e0942f194"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.656301 9708 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal/instance:
uuid: "3d4db9ea78bd4a0e9e34b805f7932e48"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "66ede35f79ba337355a8a79b5e19cb36"
server_key_iv: "ee4fe21bb0a9245adde97c1e0942f194"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.659242 9708 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:09:57.661031 9723 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:09:57.661929 9708 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:09:57.662109 9708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "3d4db9ea78bd4a0e9e34b805f7932e48"
format_stamp: "Formatted at 2026-05-01 02:09:57 on dist-test-slave-2j90"
server_key: "66ede35f79ba337355a8a79b5e19cb36"
server_key_iv: "ee4fe21bb0a9245adde97c1e0942f194"
server_key_version: "encryptionkey@0"
I20260501 02:09:57.662202 9708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:09:57.674851 9708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:09:57.677155 9708 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:09:57.677342 9708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:09:57.684307 9708 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.252:44467
I20260501 02:09:57.684309 9775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.252:44467 every 8 connection(s)
I20260501 02:09:57.685221 9708 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
I20260501 02:09:57.687722 9776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:09:57.689555 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9708
I20260501 02:09:57.689688 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1777600936179445-18819-0/minicluster-data/master-2/wal/instance
I20260501 02:09:57.689998 18819 external_mini_cluster.cc:1468] Setting key 4cc7c975539019597f828db17433e11c
I20260501 02:09:57.691870 9776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:09:57.702025 9626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.693536 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.18.96.254:38455)
0501 02:09:57.693693 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.693697 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.693943 (+ 246us) server_negotiation.cc:408] Connection header received
0501 02:09:57.694103 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.694105 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.694150 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.694219 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.695166 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.695580 (+ 414us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.696225 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.696372 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.698301 (+ 1929us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.698324 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.698328 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.698361 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.700097 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.700638 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.700642 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.700645 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.700697 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.701017 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.701020 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.701021 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.701190 (+ 169us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.701289 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.701738 (+ 449us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.701849 (+ 111us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:09:57.702538 9779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.691359 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.18.96.252:44467)
0501 02:09:57.691756 (+ 397us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.691765 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.691794 (+ 29us) server_negotiation.cc:408] Connection header received
0501 02:09:57.692370 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.692388 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.692704 (+ 316us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.692985 (+ 281us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.694026 (+ 1041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.695175 (+ 1149us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.695825 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.696094 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.698873 (+ 2779us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.698893 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.698905 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.698937 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.700721 (+ 1784us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.701139 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.701145 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.701146 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.701188 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.701440 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.701442 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.701444 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.701601 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.701788 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.702061 (+ 273us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.702187 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260501 02:09:57.702540 9780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.693479 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:38455 (local address 127.0.0.1:52110)
0501 02:09:57.693826 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.693861 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.694010 (+ 149us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.694293 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.694298 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.694713 (+ 415us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.695005 (+ 292us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.695015 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.695694 (+ 679us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.695698 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.696093 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.696103 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.696248 (+ 145us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.696717 (+ 469us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.696733 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.698173 (+ 1440us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.700237 (+ 2064us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.700243 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.700248 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.700521 (+ 273us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.700813 (+ 292us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.700815 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.700817 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.700919 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.701301 (+ 382us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.701307 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.701520 (+ 213us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.701692 (+ 172us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.701888 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":110,"threads_started":1}
I20260501 02:09:57.703552 9776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:09:57.710956 9780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.704009 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41649 (local address 127.0.0.1:35274)
0501 02:09:57.704149 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.704163 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.704239 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.704611 (+ 372us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.704613 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.704743 (+ 130us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.704877 (+ 134us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.704881 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.705546 (+ 665us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.705551 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.705882 (+ 331us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.705887 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.706036 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.706462 (+ 426us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.706473 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.707843 (+ 1370us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.709599 (+ 1756us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.709602 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.709604 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.709814 (+ 210us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.710086 (+ 272us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.710089 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.710091 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.710137 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.710486 (+ 349us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.710489 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.710562 (+ 73us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.710681 (+ 119us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.710798 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260501 02:09:57.710980 9703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.704129 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.18.96.253:41649)
0501 02:09:57.704320 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.704324 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.704339 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:09:57.704383 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.704387 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.704437 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.704522 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.705003 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.705440 (+ 437us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.706019 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.706226 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.707963 (+ 1737us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.707984 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.707986 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.708008 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.709475 (+ 1467us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.709899 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.709902 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.709904 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.709993 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.710248 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.710251 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.710253 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.710394 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.710493 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.710730 (+ 237us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.710843 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260501 02:09:57.711751 9626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.704502 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.18.96.254:38455)
0501 02:09:57.704616 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.704620 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.704642 (+ 22us) server_negotiation.cc:408] Connection header received
0501 02:09:57.704715 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.704718 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.704762 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.704841 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.705484 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.705892 (+ 408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.706618 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.706826 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.709057 (+ 2231us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.709077 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.709079 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.709099 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.710310 (+ 1211us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.710681 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.710696 (+ 15us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.710698 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.710744 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.710986 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.710990 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.710992 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.711156 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.711308 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.711498 (+ 190us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.711595 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260501 02:09:57.711979 9776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:09:57.712401 9622 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 3
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:09:57.720860 9780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.712425 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44934)
0501 02:09:57.712747 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.712762 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.712827 (+ 65us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.713394 (+ 567us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.713398 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.713579 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.713791 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.713798 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.714781 (+ 983us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.714785 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.715232 (+ 447us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.715239 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.715360 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.715875 (+ 515us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.715911 (+ 36us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.717396 (+ 1485us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.719451 (+ 2055us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.719454 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.719460 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.719744 (+ 284us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.719987 (+ 243us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.719989 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.719991 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.720037 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.720403 (+ 366us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.720409 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.720480 (+ 71us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.720596 (+ 116us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.720697 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260501 02:09:57.721287 9625 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.712729 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44944)
0501 02:09:57.713110 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.713125 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.713209 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.713511 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.713514 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.713643 (+ 129us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.713776 (+ 133us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.713780 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.714487 (+ 707us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.714492 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.714898 (+ 406us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.714905 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.715007 (+ 102us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.715438 (+ 431us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.715455 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.716839 (+ 1384us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.719976 (+ 3137us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.719982 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.719986 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.720189 (+ 203us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.720422 (+ 233us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.720425 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.720427 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.720479 (+ 52us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.720872 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.720875 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.720966 (+ 91us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.721070 (+ 104us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.721182 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260501 02:09:57.721364 9781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.712903 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.18.96.252:44467)
0501 02:09:57.713176 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.713180 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.713193 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:09:57.713223 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.713226 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.713267 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.713357 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.713902 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.714677 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.715550 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.715765 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.717518 (+ 1753us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.717539 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.717541 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.717560 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.719328 (+ 1768us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.719849 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.719852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.719854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.719894 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.720125 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.720133 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.720134 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.720284 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.720380 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.721096 (+ 716us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.721195 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
I20260501 02:09:57.721800 9779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.712767 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.18.96.252:44467)
0501 02:09:57.712884 (+ 117us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.712888 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.713175 (+ 287us) server_negotiation.cc:408] Connection header received
0501 02:09:57.713302 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.713307 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.713350 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.713423 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.713889 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.714382 (+ 493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.715007 (+ 625us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.715467 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.716977 (+ 1510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.716996 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.717002 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.717045 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.719868 (+ 2823us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.720284 (+ 416us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.720288 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.720289 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.720326 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.720606 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.720612 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.720614 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.720768 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.720879 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.721611 (+ 732us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.721709 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260501 02:09:57.722491 9703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.713076 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.18.96.253:41649)
0501 02:09:57.713190 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.713193 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.713446 (+ 253us) server_negotiation.cc:408] Connection header received
0501 02:09:57.713610 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.713613 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.713653 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.713702 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.714631 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.715178 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.715962 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.716163 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.718415 (+ 2252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.718434 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.718438 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.718472 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.720192 (+ 1720us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.720766 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.720769 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.720771 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.720815 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.721097 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.721102 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.721104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.721248 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.721416 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.722242 (+ 826us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.722325 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260501 02:09:57.723343 9699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } attempt: 3
I20260501 02:09:57.724772 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: Bootstrap starting.
I20260501 02:09:57.724771 9776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: Bootstrap starting.
May 01 02:09:57 dist-test-slave-2j90 krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601397, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:09:57.727461 9776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:57.727463 9622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:57.728358 9622 log.cc:826] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:57.728642 9776 log.cc:826] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:57.730567 9702 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.723757 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:44467 (local address 127.0.0.1:44954)
0501 02:09:57.723875 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:09:57.723889 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:09:57.723967 (+ 78us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:09:57.724268 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:09:57.724272 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:09:57.724420 (+ 148us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:09:57.724557 (+ 137us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.724561 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.725176 (+ 615us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.725181 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:09:57.725508 (+ 327us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:09:57.725513 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.725635 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.726109 (+ 474us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:09:57.726124 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:09:57.727235 (+ 1111us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:09:57.729169 (+ 1934us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.729172 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.729174 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.729386 (+ 212us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.729663 (+ 277us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:09:57.729666 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:09:57.729668 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:09:57.729721 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:09:57.730154 (+ 433us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:09:57.730157 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:09:57.730229 (+ 72us) client_negotiation.cc:770] Sending connection context
0501 02:09:57.730324 (+ 95us) client_negotiation.cc:241] Negotiation successful
0501 02:09:57.730431 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260501 02:09:57.730652 9776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: No bootstrap required, opened a new log
I20260501 02:09:57.731005 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:09:57.731138 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: No bootstrap required, opened a new log
I20260501 02:09:57.731180 9779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.723930 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.18.96.252:44467)
0501 02:09:57.724093 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.724096 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.724105 (+ 9us) server_negotiation.cc:408] Connection header received
0501 02:09:57.724129 (+ 24us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.724131 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.724161 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.724254 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.724674 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.725077 (+ 403us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.725614 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.726116 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.727349 (+ 1233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.727367 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.727369 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.727394 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.729054 (+ 1660us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.729503 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.729506 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.729507 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.729548 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.729830 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.729833 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.729835 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.730019 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.730111 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.730965 (+ 854us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.731061 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260501 02:09:57.733188 9776 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.733361 9776 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4db9ea78bd4a0e9e34b805f7932e48, State: Initialized, Role: FOLLOWER
I20260501 02:09:57.733649 9776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.734095 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.734262 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.734222 9622 raft_consensus.cc:359] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.734421 9622 raft_consensus.cc:740] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49648785c78474b9864aa728c098774, State: Initialized, Role: FOLLOWER
I20260501 02:09:57.734642 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: Bootstrap starting.
I20260501 02:09:57.734700 9622 consensus_queue.cc:260] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.734861 9776 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:57.735699 9622 sys_catalog.cc:565] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:57.737761 9596 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e49648785c78474b9864aa728c098774"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52116
I20260501 02:09:57.737963 9596 raft_consensus.cc:493] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260501 02:09:57.738190 9596 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:57.739363 9596 raft_consensus.cc:515] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
W20260501 02:09:57.739758 9801 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:57.739813 9801 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:57.738405 9708 master_runner.cc:186] Error getting master registration for 127.18.96.254:38455: OK, instance_id { permanent_uuid: "e49648785c78474b9864aa728c098774" instance_seqno: 1777601397251439 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 02:09:57.741011 9708 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:09:57.740973 9750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e49648785c78474b9864aa728c098774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48"
I20260501 02:09:57.741118 9699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: Neither blocks nor log segments found. Creating new log.
I20260501 02:09:57.741144 9750 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:57.742017 9699 log.cc:826] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: Log is configured to *not* fsync() on all Append() calls
I20260501 02:09:57.740239 9596 leader_election.cc:290] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [CANDIDATE]: Term 1 election: Requested vote from peers 5cfc1426daf5406f8110b8e077366fdf (127.18.96.253:41649), 3d4db9ea78bd4a0e9e34b805f7932e48 (127.18.96.252:44467)
I20260501 02:09:57.742930 9750 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49648785c78474b9864aa728c098774 in term 1.
I20260501 02:09:57.743362 9571 leader_election.cc:304] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4db9ea78bd4a0e9e34b805f7932e48, e49648785c78474b9864aa728c098774; no voters:
I20260501 02:09:57.743510 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.743615 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.743693 9785 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:09:57.743777 9785 raft_consensus.cc:697] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [term 1 LEADER]: Becoming Leader. State: Replica: e49648785c78474b9864aa728c098774, State: Running, Role: LEADER
I20260501 02:09:57.744069 9785 consensus_queue.cc:237] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.745697 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49648785c78474b9864aa728c098774. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.745808 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:57.744736 9554 master_runner.cc:186] Error getting master registration for 127.18.96.253:41649: OK, instance_id { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" instance_seqno: 1777601397477409 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 02:09:57.747563 9554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260501 02:09:57.747817 9575 tablet.cc:2404] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260501 02:09:57.750442 9808 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:57.750509 9808 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:57.750583 9817 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:09:57.751233 9817 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:09:57.755718 9750 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [term 1 FOLLOWER]: Refusing update from remote peer e49648785c78474b9864aa728c098774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:57.756426 9814 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:57.757423 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: No bootstrap required, opened a new log
I20260501 02:09:57.759658 9626 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.747956 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.18.96.254:38455)
0501 02:09:57.748063 (+ 107us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.748067 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.748734 (+ 667us) server_negotiation.cc:408] Connection header received
0501 02:09:57.749018 (+ 284us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.749022 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.749071 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.749123 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.751037 (+ 1914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.751744 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.752980 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.753194 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.755641 (+ 2447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.755669 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.755672 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.755697 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.757503 (+ 1806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.758179 (+ 676us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.758183 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.758185 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.758235 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.758663 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.758667 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.758668 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.758828 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.758931 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.759397 (+ 466us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.759500 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:09:57.760747 9699 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.760946 9699 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfc1426daf5406f8110b8e077366fdf, State: Initialized, Role: FOLLOWER
I20260501 02:09:57.761265 9699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } }
I20260501 02:09:57.762046 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49648785c78474b9864aa728c098774. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.762176 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.762801 9699 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:09:57.762867 9673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e49648785c78474b9864aa728c098774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5cfc1426daf5406f8110b8e077366fdf"
I20260501 02:09:57.763036 9672 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:09:57.764302 9672 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [term 1 FOLLOWER]: Refusing update from remote peer e49648785c78474b9864aa728c098774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:09:57.764942 9814 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:09:57.766757 9673 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49648785c78474b9864aa728c098774 in term 1.
I20260501 02:09:57.769791 9703 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.750317 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.18.96.253:41649)
0501 02:09:57.750471 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.750475 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.761240 (+ 10765us) server_negotiation.cc:408] Connection header received
0501 02:09:57.761505 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.761511 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.761567 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.761619 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.763159 (+ 1540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.763833 (+ 674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.764813 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.765009 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.766554 (+ 1545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.766568 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.766571 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.766598 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.768207 (+ 1609us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.768638 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.768644 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.768648 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.768696 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.768963 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.768967 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.768969 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.769131 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.769231 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.769545 (+ 314us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.769647 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260501 02:09:57.773859 9779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:57.748648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.18.96.252:44467)
0501 02:09:57.748756 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.748760 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.753007 (+ 4247us) server_negotiation.cc:408] Connection header received
0501 02:09:57.756282 (+ 3275us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.756285 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.756337 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.756391 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.757333 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.757982 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:57.758770 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:57.758937 (+ 167us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:57.770058 (+ 11121us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:09:57.770077 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:09:57.770081 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:09:57.770110 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:09:57.771964 (+ 1854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.772375 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.772378 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.772392 (+ 14us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.772438 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:09:57.772676 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:09:57.772679 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:09:57.772681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:09:57.772830 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:09:57.772943 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:57.773606 (+ 663us) server_negotiation.cc:300] Negotiation successful
0501 02:09:57.773715 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260501 02:09:57.774833 9836 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:09:57.774914 9836 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:09:57.775167 9825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.775250 9825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.775328 9825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e49648785c78474b9864aa728c098774. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.775383 9825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.775835 9785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.775945 9785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:57.776253 9784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.776348 9784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.776546 9822 mvcc.cc:204] Tried to move back new op lower bound from 7281055325200732160 to 7281055325163069440. Current Snapshot: MvccSnapshot[applied={T|T < 7281055325200732160}]
I20260501 02:09:57.778034 9823 mvcc.cc:204] Tried to move back new op lower bound from 7281055325200732160 to 7281055325163069440. Current Snapshot: MvccSnapshot[applied={T|T < 7281055325200732160}]
I20260501 02:09:57.782259 9814 sys_catalog.cc:455] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.782387 9814 sys_catalog.cc:458] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774 [sys.catalog]: This master's current role is: LEADER
I20260501 02:09:57.783296 9837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e49648785c78474b9864aa728c098774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49648785c78474b9864aa728c098774" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 38455 } } peers { permanent_uuid: "5cfc1426daf5406f8110b8e077366fdf" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41649 } } peers { permanent_uuid: "3d4db9ea78bd4a0e9e34b805f7932e48" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 44467 } } }
I20260501 02:09:57.783403 9837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:09:57.784462 9817 catalog_manager.cc:1357] Generated new cluster ID: 3446597aa83545a3afa7f1bce8b50b4f
I20260501 02:09:57.784523 9817 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:09:57.836424 9817 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:09:57.837713 9817 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:09:57.857642 9817 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e49648785c78474b9864aa728c098774: Generated new TSK 0
I20260501 02:09:57.858336 9817 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:09:57.884403 9626 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.881463 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.18.96.254:38455)
0501 02:09:57.881591 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.881595 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.883369 (+ 1774us) server_negotiation.cc:408] Connection header received
0501 02:09:57.883561 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.883564 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.883618 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.883671 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.884275 (+ 604us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52140: BlockingRecv error: recv got EOF from 127.0.0.1:52140 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260501 02:09:57.885673 9779 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.882140 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.18.96.252:44467)
0501 02:09:57.882255 (+ 115us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.882259 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.884856 (+ 2597us) server_negotiation.cc:408] Connection header received
0501 02:09:57.885015 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.885019 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.885067 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.885116 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.885564 (+ 448us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44974: BlockingRecv error: recv got EOF from 127.0.0.1:44974 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260501 02:09:57.887064 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.882788 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.18.96.253:41649)
0501 02:09:57.882907 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.882911 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.886052 (+ 3141us) server_negotiation.cc:408] Connection header received
0501 02:09:57.886392 (+ 340us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.886395 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.886445 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.886501 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.886966 (+ 465us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260501 02:09:57.890954 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.889613 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.18.96.253:41649)
0501 02:09:57.889718 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.889722 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.890116 (+ 394us) server_negotiation.cc:408] Connection header received
0501 02:09:57.890308 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.890311 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.890389 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.890440 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.890867 (+ 427us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35294: BlockingRecv error: recv got EOF from 127.0.0.1:35294 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:09:57.895365 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.894121 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.18.96.253:41649)
0501 02:09:57.894216 (+ 95us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.894219 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.894629 (+ 410us) server_negotiation.cc:408] Connection header received
0501 02:09:57.894785 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.894788 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.894831 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.894875 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.895271 (+ 396us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35310: BlockingRecv error: recv got EOF from 127.0.0.1:35310 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260501 02:09:57.900549 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.899268 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.18.96.253:41649)
0501 02:09:57.899373 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.899377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.899766 (+ 389us) server_negotiation.cc:408] Connection header received
0501 02:09:57.899942 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.899946 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.899990 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.900044 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.900471 (+ 427us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35320: BlockingRecv error: recv got EOF from 127.0.0.1:35320 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260501 02:09:57.907766 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.906488 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.18.96.253:41649)
0501 02:09:57.906602 (+ 114us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.906606 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.907017 (+ 411us) server_negotiation.cc:408] Connection header received
0501 02:09:57.907185 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.907189 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.907231 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.907275 (+ 44us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.907690 (+ 415us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35324: BlockingRecv error: recv got EOF from 127.0.0.1:35324 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260501 02:09:57.920185 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.917681 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.18.96.253:41649)
0501 02:09:57.917778 (+ 97us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.917782 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.918286 (+ 504us) server_negotiation.cc:408] Connection header received
0501 02:09:57.918481 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.918485 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.918533 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.918582 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.920060 (+ 1478us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35326: BlockingRecv error: recv got EOF from 127.0.0.1:35326 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260501 02:09:57.940271 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.938911 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.18.96.253:41649)
0501 02:09:57.939047 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.939052 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.939289 (+ 237us) server_negotiation.cc:408] Connection header received
0501 02:09:57.939463 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.939467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.939532 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.939634 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.940155 (+ 521us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260501 02:09:57.976909 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:57.975110 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.18.96.253:41649)
0501 02:09:57.975239 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:57.975243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:57.975903 (+ 660us) server_negotiation.cc:408] Connection header received
0501 02:09:57.976112 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:57.976116 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:57.976173 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:57.976225 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:57.976789 (+ 564us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35342: BlockingRecv error: recv got EOF from 127.0.0.1:35342 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260501 02:09:58.044687 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:58.043599 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.18.96.253:41649)
0501 02:09:58.043781 (+ 182us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.043785 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.043872 (+ 87us) server_negotiation.cc:408] Connection header received
0501 02:09:58.043989 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.043991 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.044040 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.044114 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:58.044559 (+ 445us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35344: BlockingRecv error: recv got EOF from 127.0.0.1:35344 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260501 02:09:58.176160 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:58.174941 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.18.96.253:41649)
0501 02:09:58.175121 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.175125 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.175259 (+ 134us) server_negotiation.cc:408] Connection header received
0501 02:09:58.175441 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.175448 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.175499 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.175589 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:58.176053 (+ 464us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35356: BlockingRecv error: recv got EOF from 127.0.0.1:35356 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260501 02:09:58.435380 9703 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:09:58.434316 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.18.96.253:41649)
0501 02:09:58.434488 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.434494 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.434518 (+ 24us) server_negotiation.cc:408] Connection header received
0501 02:09:58.434705 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.434709 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.434767 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.434855 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:09:58.435257 (+ 402us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260501 02:09:58.741024 9801 catalog_manager.cc:1269] Loaded cluster ID: 3446597aa83545a3afa7f1bce8b50b4f
I20260501 02:09:58.741098 9801 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: loading cluster ID for follower catalog manager: success
I20260501 02:09:58.742815 9801 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: acquiring CA information for follower catalog manager: success
I20260501 02:09:58.743789 9801 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 3d4db9ea78bd4a0e9e34b805f7932e48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:09:58.776100 9836 catalog_manager.cc:1269] Loaded cluster ID: 3446597aa83545a3afa7f1bce8b50b4f
I20260501 02:09:58.776180 9836 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: loading cluster ID for follower catalog manager: success
I20260501 02:09:58.777782 9836 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: acquiring CA information for follower catalog manager: success
I20260501 02:09:58.778848 9836 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 5cfc1426daf5406f8110b8e077366fdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:09:58.952570 9909 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:58.949179 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.18.96.253:41649)
0501 02:09:58.949496 (+ 317us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.949500 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.949516 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:09:58.949668 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.949672 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.949784 (+ 112us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.949894 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:58.950319 (+ 425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.950851 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:58.951538 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.951716 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:58.951796 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:58.952170 (+ 374us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:58.952299 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:58.952356 (+ 57us) server_negotiation.cc:300] Negotiation successful
0501 02:09:58.952420 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260501 02:09:58.958353 9915 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:58.955578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.18.96.252:44467)
0501 02:09:58.955796 (+ 218us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.955800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.955811 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:09:58.955884 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.955886 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.955973 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.956077 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:58.956359 (+ 282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.956861 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:58.957439 (+ 578us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.957581 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:58.957672 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:58.958046 (+ 374us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:58.958127 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:58.958206 (+ 79us) server_negotiation.cc:300] Negotiation successful
0501 02:09:58.958247 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260501 02:09:58.964300 9909 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:58.961013 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.18.96.253:41649)
0501 02:09:58.961122 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.961126 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.961294 (+ 168us) server_negotiation.cc:408] Connection header received
0501 02:09:58.961485 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.961491 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.961552 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.961605 (+ 53us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:58.962159 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.962690 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:58.963520 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.963747 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:58.963793 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:58.963869 (+ 76us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:58.963938 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:58.964102 (+ 164us) server_negotiation.cc:300] Negotiation successful
0501 02:09:58.964145 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260501 02:09:58.964509 9915 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:58.961646 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.18.96.252:44467)
0501 02:09:58.961762 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.961767 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.961782 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:09:58.961861 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.961864 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.961916 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.962006 (+ 90us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:58.962382 (+ 376us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.962923 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:58.963586 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.963774 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:58.964010 (+ 236us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:58.964086 (+ 76us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:58.964151 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:58.964316 (+ 165us) server_negotiation.cc:300] Negotiation successful
0501 02:09:58.964355 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:09:58.965838 9922 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:09:58.961119 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.18.96.254:38455)
0501 02:09:58.961373 (+ 254us) server_negotiation.cc:207] Beginning negotiation
0501 02:09:58.961377 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:09:58.961392 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:09:58.961551 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:09:58.961554 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:09:58.961713 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:09:58.961823 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:09:58.963082 (+ 1259us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.963797 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:09:58.964934 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:09:58.965155 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:09:58.965246 (+ 91us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:09:58.965486 (+ 240us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:09:58.965586 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:09:58.965694 (+ 108us) server_negotiation.cc:300] Negotiation successful
0501 02:09:58.965737 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260501 02:09:58.968587 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9554
I20260501 02:09:58.975070 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9631
I20260501 02:09:58.980949 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9708
2026-05-01T02:09:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3978 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
May 01 02:09:59 dist-test-slave-2j90 krb5kdc[9926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
May 01 02:09:59 dist-test-slave-2j90 krb5kdc[9926](info): set up 2 sockets
May 01 02:09:59 dist-test-slave-2j90 krb5kdc[9926](info): commencing operation
krb5kdc: starting...
W20260501 02:10:00.950927 18819 mini_kdc.cc:121] Time spent starting KDC: real 1.948s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/test-user.keytab.
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-05-01T02:10:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-01T02:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:10:01.088603 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45541
--webserver_interface=127.18.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260501 02:10:01.190209 9942 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260501 02:10:01.190503 9942 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:10:01.190570 9942 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:10:01.190680 9942 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:10:01.194010 9942 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:10:01.194092 9942 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:10:01.194134 9942 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:10:01.194205 9942 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:10:01.194247 9942 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:10:01.198437 9942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.254:45541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.96.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:10:01.199411 9942 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:10:01.200183 9942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:10:01.205415 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.205564 9942 server_base.cc:1061] running on GCE node
W20260501 02:10:01.205375 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:10:01.205391 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.206163 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:10:01.207056 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:10:01.208238 9942 hybrid_clock.cc:648] HybridClock initialized: now 1777601401208210 us; error 39 us; skew 500 ppm
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:10:01.210853 9942 init.cc:377] Logged in from keytab as kudu/127.18.96.254@KRBTEST.COM (short username kudu)
I20260501 02:10:01.211845 9942 webserver.cc:492] Webserver started at http://127.18.96.254:40679/ using document root <none> and password file <none>
I20260501 02:10:01.212344 9942 fs_manager.cc:362] Metadata directory not provided
I20260501 02:10:01.212414 9942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:10:01.212622 9942 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:10:01.214246 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data/instance:
uuid: "0dd7c1b9694d44bb903cd27cd4681148"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "0c6c060f2b73fc1684e9da2822194646"
server_key_iv: "46ecffec566d9e8a66f6dddc21351dd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.214699 9942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance:
uuid: "0dd7c1b9694d44bb903cd27cd4681148"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "0c6c060f2b73fc1684e9da2822194646"
server_key_iv: "46ecffec566d9e8a66f6dddc21351dd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.217689 9942 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260501 02:10:01.219731 9957 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:10:01.220583 9942 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260501 02:10:01.220717 9942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal
uuid: "0dd7c1b9694d44bb903cd27cd4681148"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "0c6c060f2b73fc1684e9da2822194646"
server_key_iv: "46ecffec566d9e8a66f6dddc21351dd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.220813 9942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:10:01.235109 9942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:10:01.240273 9942 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:10:01.240460 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:10:01.247570 9942 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.254:45541
I20260501 02:10:01.247582 10009 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.254:45541 every 8 connection(s)
I20260501 02:10:01.248488 9942 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/data/info.pb
I20260501 02:10:01.251087 10010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:10:01.253780 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 9942
I20260501 02:10:01.253909 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-0/wal/instance
I20260501 02:10:01.254195 18819 external_mini_cluster.cc:1468] Setting key 26462c250159d63caec3f00208336c6c
I20260501 02:10:01.254876 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.253@KRBTEST.COM" created.
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:10:01.271417 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.257045 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45541 (local address 127.0.0.1:33644)
0501 02:10:01.257537 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.257563 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.258272 (+ 709us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.258870 (+ 598us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.258875 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.259284 (+ 409us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.259634 (+ 350us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.259676 (+ 42us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.260619 (+ 943us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.260624 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.260975 (+ 351us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.260980 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.261206 (+ 226us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.261882 (+ 676us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.261899 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.265982 (+ 4083us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.268674 (+ 2692us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.268681 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.268687 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.268969 (+ 282us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.269436 (+ 467us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.269439 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.269441 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.269521 (+ 80us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.270257 (+ 736us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.270262 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.270372 (+ 110us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.270546 (+ 174us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.270734 (+ 188us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"spinlock_wait_cycles":35072,"thread_start_us":142,"threads_started":1}
I20260501 02:10:01.271423 10013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.257044 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.18.96.254:45541)
0501 02:10:01.257436 (+ 392us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.257446 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.257669 (+ 223us) server_negotiation.cc:408] Connection header received
0501 02:10:01.258564 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.258568 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.258713 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.258870 (+ 157us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.259814 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.260514 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.261100 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.261295 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.266140 (+ 4845us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.266161 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.266170 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.266194 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.268535 (+ 2341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.269096 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.269104 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.269107 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.269195 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.269644 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.269651 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.269655 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.270131 (+ 476us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.270289 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.270664 (+ 375us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.270818 (+ 154us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260501 02:10:01.273415 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:10:01.274242 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.273942 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:47336)
0501 02:10:01.274087 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.274143 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260501 02:10:01.274461 9960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
I20260501 02:10:01.275246 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.275004 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:47348)
0501 02:10:01.275143 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.275198 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260501 02:10:01.275377 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41857: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.18.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:10:01.303323 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41857
--webserver_interface=127.18.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:10:01.324612 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } attempt: 1
I20260501 02:10:01.325544 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.325193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:47358)
0501 02:10:01.325357 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.325425 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260501 02:10:01.326305 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.326077 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:47370)
0501 02:10:01.326195 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.326241 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260501 02:10:01.326474 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41857: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
I20260501 02:10:01.394694 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } attempt: 2
I20260501 02:10:01.395572 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.395225 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:41684)
0501 02:10:01.395410 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.395477 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260501 02:10:01.396387 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.396135 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:41692)
0501 02:10:01.396267 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.396313 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260501 02:10:01.396549 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.253:41857: Network error: Client connection negotiation failed: client connection to 127.18.96.253:41857: connect: Connection refused (error 111)
W20260501 02:10:01.403615 10019 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260501 02:10:01.403914 10019 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:10:01.403986 10019 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:10:01.404101 10019 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260501 02:10:01.407480 10019 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:10:01.407562 10019 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:10:01.407614 10019 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:10:01.407657 10019 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260501 02:10:01.407718 10019 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:10:01.411839 10019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.253:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.96.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:10:01.412861 10019 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:10:01.413659 10019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:10:01.419065 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:10:01.419250 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.419404 10019 server_base.cc:1061] running on GCE node
W20260501 02:10:01.419065 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.419950 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:10:01.420773 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:10:01.421919 10019 hybrid_clock.cc:648] HybridClock initialized: now 1777601401421898 us; error 40 us; skew 500 ppm
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:10:01.424556 10019 init.cc:377] Logged in from keytab as kudu/127.18.96.253@KRBTEST.COM (short username kudu)
I20260501 02:10:01.425585 10019 webserver.cc:492] Webserver started at http://127.18.96.253:35583/ using document root <none> and password file <none>
I20260501 02:10:01.426178 10019 fs_manager.cc:362] Metadata directory not provided
I20260501 02:10:01.426249 10019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:10:01.426453 10019 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:10:01.428071 10019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data/instance:
uuid: "bd5613dcefbb4e9c9a44e761511213c4"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "f0ff87701b5fec07e3ea2e7d45a09bfd"
server_key_iv: "82e4c023e612ec9ccf5ba9f056f12cd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.428540 10019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal/instance:
uuid: "bd5613dcefbb4e9c9a44e761511213c4"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "f0ff87701b5fec07e3ea2e7d45a09bfd"
server_key_iv: "82e4c023e612ec9ccf5ba9f056f12cd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.431766 10019 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260501 02:10:01.433775 10034 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:10:01.434684 10019 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:10:01.434815 10019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal
uuid: "bd5613dcefbb4e9c9a44e761511213c4"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "f0ff87701b5fec07e3ea2e7d45a09bfd"
server_key_iv: "82e4c023e612ec9ccf5ba9f056f12cd1"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.434909 10019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:10:01.445369 10019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:10:01.447722 10019 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:10:01.447902 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:10:01.454826 10019 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.253:41857
I20260501 02:10:01.454820 10086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.253:41857 every 8 connection(s)
I20260501 02:10:01.455768 10019 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/data/info.pb
I20260501 02:10:01.458308 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 10019
I20260501 02:10:01.458396 10087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:10:01.458441 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-1/wal/instance
I20260501 02:10:01.458701 18819 external_mini_cluster.cc:1468] Setting key dad5ad5a3175c62dc9c004576f8ab1d7
I20260501 02:10:01.462342 10087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.96.252@KRBTEST.COM" created.
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:10:01.473408 10013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.463857 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.18.96.254:45541)
0501 02:10:01.464017 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.464022 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.464676 (+ 654us) server_negotiation.cc:408] Connection header received
0501 02:10:01.465358 (+ 682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.465362 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.465419 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.465529 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.466787 (+ 1258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.467264 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.467927 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.468104 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.470217 (+ 2113us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.470235 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.470237 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.470260 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.471624 (+ 1364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.472118 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.472122 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.472124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.472174 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.472442 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.472446 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.472448 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.472585 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.472714 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.473129 (+ 415us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.473239 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260501 02:10:01.473920 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.464108 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45541 (local address 127.0.0.1:54450)
0501 02:10:01.464521 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.464558 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.465175 (+ 617us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.465693 (+ 518us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.465702 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.466103 (+ 401us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.466534 (+ 431us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.466545 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.467420 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.467423 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.467816 (+ 393us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.467823 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.467963 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.468521 (+ 558us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.468538 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.470054 (+ 1516us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.471788 (+ 1734us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.471794 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.471803 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.472018 (+ 215us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.472276 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.472279 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.472280 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.472353 (+ 73us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.472715 (+ 362us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.472724 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.472909 (+ 185us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.473091 (+ 182us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.473278 (+ 187us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":134,"threads_started":1}
I20260501 02:10:01.475252 10087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:10:01.486750 10092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.475819 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.18.96.253:41857)
0501 02:10:01.476123 (+ 304us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.476128 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.476144 (+ 16us) server_negotiation.cc:408] Connection header received
0501 02:10:01.476181 (+ 37us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.476184 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.476292 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.476601 (+ 309us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.477091 (+ 490us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.477794 (+ 703us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.478429 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.478661 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.480194 (+ 1533us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.480219 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.480226 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.480270 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.484979 (+ 4709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.485480 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.485487 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.485489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.485542 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.485807 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.485809 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.485810 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.486030 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.486132 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.486467 (+ 335us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.486581 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
I20260501 02:10:01.486796 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.475711 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:41696)
0501 02:10:01.475886 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.475902 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.475987 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.476556 (+ 569us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.476559 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.476767 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.476918 (+ 151us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.476923 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.477899 (+ 976us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.477904 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.478301 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.478307 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.478465 (+ 158us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.478932 (+ 467us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.478944 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.480086 (+ 1142us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.485106 (+ 5020us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.485109 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.485111 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.485303 (+ 192us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.485640 (+ 337us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.485645 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.485648 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.485715 (+ 67us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.486133 (+ 418us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.486139 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.486266 (+ 127us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.486391 (+ 125us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.486518 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260501 02:10:01.487730 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } attempt: 3
I20260501 02:10:01.488799 10087 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:10:01.489538 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.489254 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55354)
0501 02:10:01.489397 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.489468 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260501 02:10:01.489790 10035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
I20260501 02:10:01.491420 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.491171 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55356)
0501 02:10:01.491275 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.491323 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
WARNING: no policy specified for HTTP/127.18.96.252@KRBTEST.COM; defaulting to no policy
W20260501 02:10:01.491958 10087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.96.252@KRBTEST.COM" created.
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:10:01.497743 10092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.488264 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.18.96.253:41857)
0501 02:10:01.488413 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.488419 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.488498 (+ 79us) server_negotiation.cc:408] Connection header received
0501 02:10:01.488846 (+ 348us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.488850 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.488908 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.489001 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.489601 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.490283 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.490865 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.491074 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.494665 (+ 3591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.494687 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.494693 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.494724 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.496303 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.496772 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.496774 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.496776 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.496812 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.497068 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.497070 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.497071 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.497180 (+ 109us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.497251 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.497572 (+ 321us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.497653 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260501 02:10:01.497942 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.488182 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:41698)
0501 02:10:01.488413 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.488428 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.488533 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.489042 (+ 509us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.489045 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.489252 (+ 207us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.489485 (+ 233us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.489492 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.490379 (+ 887us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.490384 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.490756 (+ 372us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.490762 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.490881 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.491351 (+ 470us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.491367 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.494538 (+ 3171us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.496420 (+ 1882us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.496427 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.496432 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.496671 (+ 239us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.496903 (+ 232us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.496909 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.496914 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.496982 (+ 68us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.497279 (+ 297us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.497286 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.497400 (+ 114us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.497527 (+ 127us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.497666 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260501 02:10:01.498697 10010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
I20260501 02:10:01.499363 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.499152 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55370)
0501 02:10:01.499272 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.499311 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260501 02:10:01.500043 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.499852 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55382)
0501 02:10:01.499967 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.500000 (+ 33us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260501 02:10:01.500231 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.18.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab.
I20260501 02:10:01.509204 18819 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
/tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:32873
--webserver_interface=127.18.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.18.96.212:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260501 02:10:01.531386 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 1
I20260501 02:10:01.532207 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.531913 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55396)
0501 02:10:01.532067 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.532128 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260501 02:10:01.533046 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.532834 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55402)
0501 02:10:01.532950 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.532996 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260501 02:10:01.533176 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
I20260501 02:10:01.541222 10087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 1
I20260501 02:10:01.541916 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.541664 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55410)
0501 02:10:01.541800 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.541850 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260501 02:10:01.542734 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.542523 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55412)
0501 02:10:01.542635 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.542682 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260501 02:10:01.542894 10087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
W20260501 02:10:01.607913 10096 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260501 02:10:01.608103 10096 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260501 02:10:01.608124 10096 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260501 02:10:01.608166 10096 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260501 02:10:01.608340 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 2
I20260501 02:10:01.609103 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.608835 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55422)
0501 02:10:01.608980 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.609033 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260501 02:10:01.609787 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.609594 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55424)
0501 02:10:01.609695 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.609733 (+ 38us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260501 02:10:01.609979 10010 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
I20260501 02:10:01.611043 10087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 2
W20260501 02:10:01.611479 10096 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260501 02:10:01.611553 10096 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260501 02:10:01.611616 10096 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260501 02:10:01.611656 10096 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260501 02:10:01.611663 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.611397 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55440)
0501 02:10:01.611517 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.611590 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260501 02:10:01.611696 10096 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260501 02:10:01.612300 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.612109 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55456)
0501 02:10:01.612210 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.612244 (+ 34us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260501 02:10:01.612421 10087 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.18.96.252:32873: Network error: Client connection negotiation failed: client connection to 127.18.96.252:32873: connect: Connection refused (error 111)
I20260501 02:10:01.615640 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.96.212:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.96.254:45541,127.18.96.253:41857,127.18.96.252:32873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/krb5kdc/kudu.keytab
--principal=kudu/127.18.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.96.252:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.96.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision aa7cc0aa9c559b87f264235eaadd03ec492277e4
build type DEBUG
built by None at 01 May 2026 01:43:13 UTC on bdcb31816ec0
build id 11684
I20260501 02:10:01.616539 10096 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260501 02:10:01.617277 10096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260501 02:10:01.622388 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.622545 10096 server_base.cc:1061] running on GCE node
W20260501 02:10:01.622427 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260501 02:10:01.622447 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260501 02:10:01.623169 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260501 02:10:01.624024 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260501 02:10:01.625201 10096 hybrid_clock.cc:648] HybridClock initialized: now 1777601401625164 us; error 49 us; skew 500 ppm
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260501 02:10:01.627830 10096 init.cc:377] Logged in from keytab as kudu/127.18.96.252@KRBTEST.COM (short username kudu)
I20260501 02:10:01.628739 10096 webserver.cc:492] Webserver started at http://127.18.96.252:37269/ using document root <none> and password file <none>
I20260501 02:10:01.629223 10096 fs_manager.cc:362] Metadata directory not provided
I20260501 02:10:01.629268 10096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260501 02:10:01.629462 10096 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260501 02:10:01.630980 10096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data/instance:
uuid: "d1e37cf437664d528192233a7082a79f"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "5d97941ec31565cb593268bda2a527ff"
server_key_iv: "f6758cbec4c5bc70772a96c2c9af26b0"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.631424 10096 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal/instance:
uuid: "d1e37cf437664d528192233a7082a79f"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "5d97941ec31565cb593268bda2a527ff"
server_key_iv: "f6758cbec4c5bc70772a96c2c9af26b0"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.634522 10096 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260501 02:10:01.636516 10111 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260501 02:10:01.637351 10096 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260501 02:10:01.637509 10096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal
uuid: "d1e37cf437664d528192233a7082a79f"
format_stamp: "Formatted at 2026-05-01 02:10:01 on dist-test-slave-2j90"
server_key: "5d97941ec31565cb593268bda2a527ff"
server_key_iv: "f6758cbec4c5bc70772a96c2c9af26b0"
server_key_version: "encryptionkey@0"
I20260501 02:10:01.637600 10096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260501 02:10:01.661459 10096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260501 02:10:01.663813 10096 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260501 02:10:01.663985 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260501 02:10:01.670809 10096 rpc_server.cc:307] RPC server started. Bound to: 127.18.96.252:32873
I20260501 02:10:01.670830 10163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.96.252:32873 every 8 connection(s)
I20260501 02:10:01.671689 10096 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/data/info.pb
I20260501 02:10:01.674167 10164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260501 02:10:01.674469 18819 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu as pid 10096
I20260501 02:10:01.674595 18819 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskQ5nb4m/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1777600936179445-18819-0/minicluster-data/master-2/wal/instance
I20260501 02:10:01.674852 18819 external_mini_cluster.cc:1468] Setting key 77bdbe34e93f4fe173184297888f0dd5
I20260501 02:10:01.678795 10164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:10:01.688127 10167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.675861 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.18.96.252:32873)
0501 02:10:01.676225 (+ 364us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.676240 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.676279 (+ 39us) server_negotiation.cc:408] Connection header received
0501 02:10:01.677613 (+ 1334us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.677631 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.677886 (+ 255us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.678178 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.679229 (+ 1051us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.680101 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.680902 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.681151 (+ 249us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.683789 (+ 2638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.683806 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.683813 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.683837 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.685487 (+ 1650us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.685961 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.685993 (+ 32us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.685998 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.686078 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.686333 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.686336 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.686337 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.686755 (+ 418us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.686948 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.687197 (+ 249us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.687409 (+ 212us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
I20260501 02:10:01.688690 10013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.680237 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.18.96.254:45541)
0501 02:10:01.680369 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.680373 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.680553 (+ 180us) server_negotiation.cc:408] Connection header received
0501 02:10:01.680681 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.680685 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.680741 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.680825 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.681688 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.682450 (+ 762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.683301 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.683516 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.685468 (+ 1952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.685491 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.685494 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.685517 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.687102 (+ 1585us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.687592 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.687595 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.687597 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.687641 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.687874 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.687876 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.687877 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.687995 (+ 118us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.688065 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.688447 (+ 382us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.688551 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260501 02:10:01.689095 10168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.680131 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.254:45541 (local address 127.0.0.1:54462)
0501 02:10:01.680441 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.680474 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.680580 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.680904 (+ 324us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.680908 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.681335 (+ 427us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.681547 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.681552 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.682573 (+ 1021us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.682579 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.683127 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.683137 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.683308 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.683770 (+ 462us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.683787 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.685343 (+ 1556us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.687223 (+ 1880us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.687229 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.687232 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.687492 (+ 260us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.687732 (+ 240us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.687735 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.687737 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.687789 (+ 52us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.688092 (+ 303us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.688097 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.688213 (+ 116us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.688820 (+ 607us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.688945 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":125,"threads_started":1}
I20260501 02:10:01.690075 10164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.254@KRBTEST.COM
I20260501 02:10:01.697486 10168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.690516 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.253:41857 (local address 127.0.0.1:41710)
0501 02:10:01.690703 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.690717 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.690777 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.691024 (+ 247us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.691026 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.691145 (+ 119us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.691293 (+ 148us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.691297 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.691990 (+ 693us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.691997 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.692338 (+ 341us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.692356 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.692438 (+ 82us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.692836 (+ 398us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.692847 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.694206 (+ 1359us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.696209 (+ 2003us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.696212 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.696213 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.696376 (+ 163us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.696634 (+ 258us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.696637 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.696639 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.696697 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.697069 (+ 372us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.697072 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.697154 (+ 82us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.697258 (+ 104us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.697339 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260501 02:10:01.697620 10092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.690631 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.18.96.253:41857)
0501 02:10:01.690777 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.690781 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.690794 (+ 13us) server_negotiation.cc:408] Connection header received
0501 02:10:01.690861 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.690864 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.690921 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.691022 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.691415 (+ 393us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.691887 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.692462 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.693080 (+ 618us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.694333 (+ 1253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.694356 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.694362 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.694402 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.696088 (+ 1686us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.696473 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.696479 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.696481 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.696531 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.696788 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.696790 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.696792 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.696954 (+ 162us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.697059 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.697331 (+ 272us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.697464 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260501 02:10:01.698176 10013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.690206 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.18.96.254:45541)
0501 02:10:01.690315 (+ 109us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.690319 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.690517 (+ 198us) server_negotiation.cc:408] Connection header received
0501 02:10:01.690626 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.690632 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.690678 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.690761 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.691529 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.691923 (+ 394us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.692552 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.692695 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.695120 (+ 2425us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.695140 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.695145 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.695174 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.696514 (+ 1340us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.696926 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.696930 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.696932 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.696974 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.697203 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.697205 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.697207 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.697370 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.697560 (+ 190us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.697886 (+ 326us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.698019 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:10:01.698591 10164 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.252@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.96.253@KRBTEST.COM
I20260501 02:10:01.703562 10087 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 3
I20260501 02:10:01.706617 10167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.699188 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.18.96.252:32873)
0501 02:10:01.699300 (+ 112us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.699304 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.699453 (+ 149us) server_negotiation.cc:408] Connection header received
0501 02:10:01.699536 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.699539 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.699581 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.699668 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.700086 (+ 418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.700665 (+ 579us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.701251 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.701463 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.703240 (+ 1777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.703256 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.703259 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.703284 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.705091 (+ 1807us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.705549 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.705554 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.705555 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.705599 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.705842 (+ 243us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.705845 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.705846 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.706021 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.706092 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.706385 (+ 293us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.706488 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:10:01.706995 10168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.699120 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55480)
0501 02:10:01.699369 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.699383 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.699462 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.699700 (+ 238us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.699704 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.699840 (+ 136us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.699979 (+ 139us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.699983 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.700762 (+ 779us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.700765 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.701149 (+ 384us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.701155 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.701236 (+ 81us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.701817 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.701829 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.703119 (+ 1290us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.705207 (+ 2088us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.705210 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.705213 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.705442 (+ 229us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.705706 (+ 264us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.705709 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.705711 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.705754 (+ 43us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.706154 (+ 400us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.706159 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.706228 (+ 69us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.706762 (+ 534us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.706866 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260501 02:10:01.707546 10092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.699252 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.18.96.253:41857)
0501 02:10:01.699371 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.699383 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.699398 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:10:01.699506 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.699512 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.699557 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.699622 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.700409 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.700871 (+ 462us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.701485 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.701689 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.703891 (+ 2202us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.703910 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.703915 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.703944 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.705738 (+ 1794us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.706193 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.706196 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.706198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.706239 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.706507 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.706511 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.706513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.706653 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.706836 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.707360 (+ 524us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.707451 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.253@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:10:01.709162 10010 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } attempt: 3
I20260501 02:10:01.710496 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: Bootstrap starting.
I20260501 02:10:01.713135 10090 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.703923 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55484)
0501 02:10:01.704057 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.704071 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.704147 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.705165 (+ 1018us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.705168 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.705479 (+ 311us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.705698 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.705708 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.707330 (+ 1622us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.707336 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.707799 (+ 463us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.707809 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.707898 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.708308 (+ 410us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.708320 (+ 12us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.709463 (+ 1143us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.711595 (+ 2132us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.711599 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.711601 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.711828 (+ 227us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.712167 (+ 339us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.712171 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.712193 (+ 22us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.712251 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.712655 (+ 404us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.712661 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.712748 (+ 87us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.712865 (+ 117us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.712974 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
May 01 02:10:01 dist-test-slave-2j90 krb5kdc[9926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777601401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.96.254@KRBTEST.COM for kudu/127.18.96.252@KRBTEST.COM
I20260501 02:10:01.713176 10164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: Neither blocks nor log segments found. Creating new log.
I20260501 02:10:01.713295 10169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.704521 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.18.96.252:32873)
0501 02:10:01.704979 (+ 458us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.704982 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.704992 (+ 10us) server_negotiation.cc:408] Connection header received
0501 02:10:01.705018 (+ 26us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.705020 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.705050 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.705150 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.705849 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.707193 (+ 1344us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.708133 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.708341 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.709588 (+ 1247us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.709603 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.709606 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.709628 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.711485 (+ 1857us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.711949 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.711954 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.711957 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.712026 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.712359 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.712362 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.712364 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.712537 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.712637 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.712933 (+ 296us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.713040 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"spinlock_wait_cycles":9216,"thread_start_us":122,"threads_started":1}
I20260501 02:10:01.714187 10164 log.cc:826] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: Log is configured to *not* fsync() on all Append() calls
I20260501 02:10:01.716138 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: No bootstrap required, opened a new log
I20260501 02:10:01.716143 10087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: Bootstrap starting.
I20260501 02:10:01.716650 18819 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260501 02:10:01.717018 10014 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.709550 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.18.96.252:32873 (local address 127.0.0.1:55490)
0501 02:10:01.709670 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0501 02:10:01.709686 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0501 02:10:01.709770 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0501 02:10:01.710174 (+ 404us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0501 02:10:01.710180 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0501 02:10:01.710352 (+ 172us) client_negotiation.cc:190] Negotiated authn=SASL
0501 02:10:01.710497 (+ 145us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.710502 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.711244 (+ 742us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.711249 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0501 02:10:01.711643 (+ 394us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0501 02:10:01.711654 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.711771 (+ 117us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.712244 (+ 473us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0501 02:10:01.712259 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0501 02:10:01.713504 (+ 1245us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0501 02:10:01.715671 (+ 2167us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.715674 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.715676 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.715885 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.716156 (+ 271us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0501 02:10:01.716162 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0501 02:10:01.716175 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0501 02:10:01.716229 (+ 54us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0501 02:10:01.716605 (+ 376us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0501 02:10:01.716608 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0501 02:10:01.716703 (+ 95us) client_negotiation.cc:770] Sending connection context
0501 02:10:01.716780 (+ 77us) client_negotiation.cc:241] Negotiation successful
0501 02:10:01.716885 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260501 02:10:01.717536 10167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.709794 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.18.96.252:32873)
0501 02:10:01.709905 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.709907 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.709916 (+ 9us) server_negotiation.cc:408] Connection header received
0501 02:10:01.709992 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.709997 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.710038 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.710625 (+ 587us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.710718 (+ 93us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.711130 (+ 412us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.711770 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.711946 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.713621 (+ 1675us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.713639 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.713642 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.713669 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.715543 (+ 1874us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.716007 (+ 464us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.716010 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.716012 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.716055 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.716339 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.716342 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.716343 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.716493 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.717194 (+ 701us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.717264 (+ 70us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.717356 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260501 02:10:01.718828 10087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: Neither blocks nor log segments found. Creating new log.
I20260501 02:10:01.719374 10087 log.cc:826] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: Log is configured to *not* fsync() on all Append() calls
I20260501 02:10:01.717891 9984 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0dd7c1b9694d44bb903cd27cd4681148"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54472
I20260501 02:10:01.719440 10164 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.719632 10164 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1e37cf437664d528192233a7082a79f, State: Initialized, Role: FOLLOWER
I20260501 02:10:01.719957 10164 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.720459 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.720638 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.721098 10164 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:10:01.721339 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: Bootstrap starting.
I20260501 02:10:01.723213 10010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: Neither blocks nor log segments found. Creating new log.
I20260501 02:10:01.723943 10010 log.cc:826] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: Log is configured to *not* fsync() on all Append() calls
I20260501 02:10:01.725853 10087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: No bootstrap required, opened a new log
W20260501 02:10:01.726364 10183 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:10:01.726498 10183 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:10:01.726980 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: No bootstrap required, opened a new log
I20260501 02:10:01.724964 10096 master_runner.cc:186] Error getting master registration for 127.18.96.254:45541: OK, instance_id { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" instance_seqno: 1777601401245391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260501 02:10:01.727538 10096 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:10:01.728827 10010 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.728994 10010 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd7c1b9694d44bb903cd27cd4681148, State: Initialized, Role: FOLLOWER
I20260501 02:10:01.730293 10010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.730835 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.731087 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.731370 10010 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:10:01.731452 9984 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260501 02:10:01.731540 9984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:10:01.731889 10087 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.732038 10087 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5613dcefbb4e9c9a44e761511213c4, State: Initialized, Role: FOLLOWER
I20260501 02:10:01.732334 10087 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.732653 9984 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.733457 10087 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260501 02:10:01.733506 9984 leader_election.cc:290] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [CANDIDATE]: Term 1 election: Requested vote from peers bd5613dcefbb4e9c9a44e761511213c4 (127.18.96.253:41857), d1e37cf437664d528192233a7082a79f (127.18.96.252:32873)
I20260501 02:10:01.734571 10061 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd7c1b9694d44bb903cd27cd4681148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd5613dcefbb4e9c9a44e761511213c4"
I20260501 02:10:01.734747 10061 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:10:01.735975 10061 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd7c1b9694d44bb903cd27cd4681148 in term 1.
I20260501 02:10:01.736370 10138 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0dd7c1b9694d44bb903cd27cd4681148" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1e37cf437664d528192233a7082a79f"
I20260501 02:10:01.736410 9960 leader_election.cc:304] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd7c1b9694d44bb903cd27cd4681148, bd5613dcefbb4e9c9a44e761511213c4; no voters:
I20260501 02:10:01.736553 10138 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [term 0 FOLLOWER]: Advancing to term 1
I20260501 02:10:01.737692 10138 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd7c1b9694d44bb903cd27cd4681148 in term 1.
W20260501 02:10:01.741624 9963 tablet.cc:2404] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260501 02:10:01.741494 10186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 1 FOLLOWER]: Leader election won for term 1
I20260501 02:10:01.742321 10019 master_runner.cc:263] No leader master found from master bd5613dcefbb4e9c9a44e761511213c4
I20260501 02:10:01.742374 10019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:10:01.742368 10186 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd7c1b9694d44bb903cd27cd4681148, State: Running, Role: LEADER
I20260501 02:10:01.742666 10186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } }
I20260501 02:10:01.742841 9942 master_runner.cc:263] No leader master found from master 0dd7c1b9694d44bb903cd27cd4681148
I20260501 02:10:01.742899 9942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260501 02:10:01.744545 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd7c1b9694d44bb903cd27cd4681148. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.744714 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: This master's current role is: LEADER
I20260501 02:10:01.746395 10219 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260501 02:10:01.747210 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.747327 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.748713 10013 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.739553 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.18.96.254:45541)
0501 02:10:01.739680 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.739686 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.740546 (+ 860us) server_negotiation.cc:408] Connection header received
0501 02:10:01.740742 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.740745 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.740792 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.740845 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.742201 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.742867 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.743486 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.743678 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.745176 (+ 1498us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.745192 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.745195 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.745219 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.746913 (+ 1694us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.747376 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.747382 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.747397 (+ 15us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.747451 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.747763 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.747767 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.747770 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.747944 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.748054 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.748437 (+ 383us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.748535 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260501 02:10:01.749801 10219 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260501 02:10:01.753142 10092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.740317 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.18.96.253:41857)
0501 02:10:01.740529 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.740533 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.744135 (+ 3602us) server_negotiation.cc:408] Connection header received
0501 02:10:01.744328 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.744333 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.744385 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.744451 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.745416 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.746157 (+ 741us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.747027 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.747218 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.748676 (+ 1458us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.748692 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.748695 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.748722 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.750717 (+ 1995us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.751838 (+ 1121us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.751843 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.751846 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.751897 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.752190 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.752196 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.752200 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.752385 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.752490 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.752867 (+ 377us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.752968 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260501 02:10:01.754837 10167 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:01.741286 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.18.96.252:32873)
0501 02:10:01.741594 (+ 308us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.741598 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.746654 (+ 5056us) server_negotiation.cc:408] Connection header received
0501 02:10:01.746814 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.746818 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.746863 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.746909 (+ 46us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.748175 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.748831 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:01.749625 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:01.749846 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:01.751446 (+ 1600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0501 02:10:01.751468 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0501 02:10:01.751471 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0501 02:10:01.751498 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0501 02:10:01.753216 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.753671 (+ 455us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.753676 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.753681 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.753735 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0501 02:10:01.754031 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0501 02:10:01.754034 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0501 02:10:01.754036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0501 02:10:01.754189 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0501 02:10:01.754303 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:01.754648 (+ 345us) server_negotiation.cc:300] Negotiation successful
0501 02:10:01.754739 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
W20260501 02:10:01.756588 10209 catalog_manager.cc:1568] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260501 02:10:01.756677 10209 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260501 02:10:01.760789 10138 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [term 1 FOLLOWER]: Refusing update from remote peer 0dd7c1b9694d44bb903cd27cd4681148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:10:01.761417 10217 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:10:01.765262 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd7c1b9694d44bb903cd27cd4681148. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.765362 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.766412 10061 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [term 1 FOLLOWER]: Refusing update from remote peer 0dd7c1b9694d44bb903cd27cd4681148: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260501 02:10:01.766989 10217 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260501 02:10:01.772579 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd7c1b9694d44bb903cd27cd4681148. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.772702 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.776821 10217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.776932 10217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: This master's current role is: LEADER
I20260501 02:10:01.778124 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.778219 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148 [sys.catalog]: This master's current role is: LEADER
I20260501 02:10:01.778132 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.778254 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.779214 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0dd7c1b9694d44bb903cd27cd4681148" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd7c1b9694d44bb903cd27cd4681148" member_type: VOTER last_known_addr { host: "127.18.96.254" port: 45541 } } peers { permanent_uuid: "bd5613dcefbb4e9c9a44e761511213c4" member_type: VOTER last_known_addr { host: "127.18.96.253" port: 41857 } } peers { permanent_uuid: "d1e37cf437664d528192233a7082a79f" member_type: VOTER last_known_addr { host: "127.18.96.252" port: 32873 } } }
I20260501 02:10:01.779309 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f [sys.catalog]: This master's current role is: FOLLOWER
I20260501 02:10:01.780403 10223 mvcc.cc:204] Tried to move back new op lower bound from 7281055341606432768 to 7281055341541957632. Current Snapshot: MvccSnapshot[applied={T|T < 7281055341606432768}]
I20260501 02:10:01.780613 10225 mvcc.cc:204] Tried to move back new op lower bound from 7281055341606432768 to 7281055341541957632. Current Snapshot: MvccSnapshot[applied={T|T < 7281055341606432768}]
I20260501 02:10:01.782332 10219 catalog_manager.cc:1357] Generated new cluster ID: e057f9157b284db4b87c7ab38a78deed
I20260501 02:10:01.782411 10219 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260501 02:10:01.804121 10219 catalog_manager.cc:1380] Generated new certificate authority record
I20260501 02:10:01.805567 10219 catalog_manager.cc:1514] Loading token signing keys...
I20260501 02:10:01.824657 10219 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0dd7c1b9694d44bb903cd27cd4681148: Generated new TSK 0
I20260501 02:10:01.825438 10219 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260501 02:10:01.847263 10013 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.844576 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54498 (local address 127.18.96.254:45541)
0501 02:10:01.844711 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.844715 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.845885 (+ 1170us) server_negotiation.cc:408] Connection header received
0501 02:10:01.846429 (+ 544us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.846433 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.846490 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.846546 (+ 56us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.847136 (+ 590us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54498: BlockingRecv error: recv got EOF from 127.0.0.1:54498 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260501 02:10:01.849272 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.845261 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.18.96.253:41857)
0501 02:10:01.845391 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.845395 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.848031 (+ 2636us) server_negotiation.cc:408] Connection header received
0501 02:10:01.848539 (+ 508us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.848543 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.848595 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.848646 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.849160 (+ 514us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41738: BlockingRecv error: recv got EOF from 127.0.0.1:41738 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260501 02:10:01.850433 10167 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.846292 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.18.96.252:32873)
0501 02:10:01.846432 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.846437 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.848445 (+ 2008us) server_negotiation.cc:408] Connection header received
0501 02:10:01.849744 (+ 1299us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.849748 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.849797 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.849847 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.850313 (+ 466us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55518: BlockingRecv error: recv got EOF from 127.0.0.1:55518 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:10:01.854924 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.853431 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.18.96.253:41857)
0501 02:10:01.853537 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.853541 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.853998 (+ 457us) server_negotiation.cc:408] Connection header received
0501 02:10:01.854214 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.854218 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.854287 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.854338 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.854827 (+ 489us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41744: BlockingRecv error: recv got EOF from 127.0.0.1:41744 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260501 02:10:01.859826 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.858396 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.18.96.253:41857)
0501 02:10:01.858501 (+ 105us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.858505 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.858979 (+ 474us) server_negotiation.cc:408] Connection header received
0501 02:10:01.859177 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.859181 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.859230 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.859279 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.859735 (+ 456us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41760: BlockingRecv error: recv got EOF from 127.0.0.1:41760 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260501 02:10:01.865159 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.863724 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.18.96.253:41857)
0501 02:10:01.863824 (+ 100us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.863830 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.864329 (+ 499us) server_negotiation.cc:408] Connection header received
0501 02:10:01.864517 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.864521 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.864570 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.864620 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.865069 (+ 449us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41766: BlockingRecv error: recv got EOF from 127.0.0.1:41766 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260501 02:10:01.872676 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.871209 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.18.96.253:41857)
0501 02:10:01.871339 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.871343 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.871841 (+ 498us) server_negotiation.cc:408] Connection header received
0501 02:10:01.872011 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.872015 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.872065 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.872115 (+ 50us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.872580 (+ 465us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41770: BlockingRecv error: recv got EOF from 127.0.0.1:41770 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260501 02:10:01.884402 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.882841 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.18.96.253:41857)
0501 02:10:01.882952 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.882955 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.883472 (+ 517us) server_negotiation.cc:408] Connection header received
0501 02:10:01.883660 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.883664 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.883720 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.883772 (+ 52us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.884288 (+ 516us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41776: BlockingRecv error: recv got EOF from 127.0.0.1:41776 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260501 02:10:01.904893 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.903009 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.18.96.253:41857)
0501 02:10:01.903149 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.903154 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.903791 (+ 637us) server_negotiation.cc:408] Connection header received
0501 02:10:01.904028 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.904035 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.904110 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.904175 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.904775 (+ 600us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41778: BlockingRecv error: recv got EOF from 127.0.0.1:41778 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260501 02:10:01.941551 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:01.939705 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.18.96.253:41857)
0501 02:10:01.939840 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:01.939844 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:01.940431 (+ 587us) server_negotiation.cc:408] Connection header received
0501 02:10:01.940681 (+ 250us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:01.940686 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:01.940764 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:01.940833 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:01.941426 (+ 593us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41784: BlockingRecv error: recv got EOF from 127.0.0.1:41784 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260501 02:10:02.009507 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:02.008175 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.18.96.253:41857)
0501 02:10:02.008334 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.008338 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.008588 (+ 250us) server_negotiation.cc:408] Connection header received
0501 02:10:02.008759 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.008762 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.008822 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.008912 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:02.009356 (+ 444us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41796: BlockingRecv error: recv got EOF from 127.0.0.1:41796 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260501 02:10:02.140651 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:02.139638 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.18.96.253:41857)
0501 02:10:02.139778 (+ 140us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.139781 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.139882 (+ 101us) server_negotiation.cc:408] Connection header received
0501 02:10:02.139983 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.139986 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.140039 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.140123 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:02.140507 (+ 384us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41812: BlockingRecv error: recv got EOF from 127.0.0.1:41812 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260501 02:10:02.399832 10092 negotiation.cc:336] Failed RPC negotiation. Trace:
0501 02:10:02.398868 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.18.96.253:41857)
0501 02:10:02.399005 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.399008 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.399023 (+ 15us) server_negotiation.cc:408] Connection header received
0501 02:10:02.399151 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.399154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.399206 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.399288 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0501 02:10:02.399692 (+ 404us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41814: BlockingRecv error: recv got EOF from 127.0.0.1:41814 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260501 02:10:02.727735 10183 catalog_manager.cc:1269] Loaded cluster ID: e057f9157b284db4b87c7ab38a78deed
I20260501 02:10:02.727819 10183 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: loading cluster ID for follower catalog manager: success
I20260501 02:10:02.729420 10183 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: acquiring CA information for follower catalog manager: success
I20260501 02:10:02.730500 10183 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d1e37cf437664d528192233a7082a79f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:10:02.757762 10209 catalog_manager.cc:1269] Loaded cluster ID: e057f9157b284db4b87c7ab38a78deed
I20260501 02:10:02.757825 10209 catalog_manager.cc:1562] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: loading cluster ID for follower catalog manager: success
I20260501 02:10:02.759353 10209 catalog_manager.cc:1584] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: acquiring CA information for follower catalog manager: success
I20260501 02:10:02.760306 10209 catalog_manager.cc:1612] T 00000000000000000000000000000000 P bd5613dcefbb4e9c9a44e761511213c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260501 02:10:02.917656 10295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:02.914164 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.18.96.253:41857)
0501 02:10:02.914463 (+ 299us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.914469 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.914567 (+ 98us) server_negotiation.cc:408] Connection header received
0501 02:10:02.914760 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.914764 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.914883 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.915002 (+ 119us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:10:02.915387 (+ 385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.915845 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:02.916505 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.916684 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:02.916762 (+ 78us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:02.917169 (+ 407us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:02.917291 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:02.917405 (+ 114us) server_negotiation.cc:300] Negotiation successful
0501 02:10:02.917471 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260501 02:10:02.923497 10301 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:02.920514 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.18.96.252:32873)
0501 02:10:02.920734 (+ 220us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.920737 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.920748 (+ 11us) server_negotiation.cc:408] Connection header received
0501 02:10:02.920820 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.920823 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.920903 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.920987 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:10:02.921252 (+ 265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.921743 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:02.922474 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.922661 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:02.922733 (+ 72us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:02.923120 (+ 387us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:02.923235 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:02.923300 (+ 65us) server_negotiation.cc:300] Negotiation successful
0501 02:10:02.923345 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260501 02:10:02.929729 10301 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:02.926490 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.18.96.252:32873)
0501 02:10:02.926601 (+ 111us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.926603 (+ 2us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.927035 (+ 432us) server_negotiation.cc:408] Connection header received
0501 02:10:02.927160 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.927164 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.927216 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.927308 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:10:02.927736 (+ 428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.928222 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:02.928906 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.929097 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:02.929138 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:02.929208 (+ 70us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:02.929297 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:02.929507 (+ 210us) server_negotiation.cc:300] Negotiation successful
0501 02:10:02.929562 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260501 02:10:02.930274 10308 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:02.926383 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.18.96.254:45541)
0501 02:10:02.926767 (+ 384us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.926772 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.926786 (+ 14us) server_negotiation.cc:408] Connection header received
0501 02:10:02.926866 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.926869 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.926948 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.927056 (+ 108us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:10:02.927467 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.928078 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:02.929081 (+ 1003us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.929285 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:02.929381 (+ 96us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:02.929691 (+ 310us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:02.929780 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:02.930061 (+ 281us) server_negotiation.cc:300] Negotiation successful
0501 02:10:02.930121 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":84,"threads_started":1}
I20260501 02:10:02.930903 10295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0501 02:10:02.926640 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.18.96.253:41857)
0501 02:10:02.926766 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0501 02:10:02.926770 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0501 02:10:02.927552 (+ 782us) server_negotiation.cc:408] Connection header received
0501 02:10:02.927591 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0501 02:10:02.927593 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0501 02:10:02.927640 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0501 02:10:02.927725 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
0501 02:10:02.928715 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.929263 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0501 02:10:02.930188 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0501 02:10:02.930372 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0501 02:10:02.930423 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0501 02:10:02.930495 (+ 72us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0501 02:10:02.930565 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0501 02:10:02.930732 (+ 167us) server_negotiation.cc:300] Negotiation successful
0501 02:10:02.930782 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260501 02:10:02.933451 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 9942
I20260501 02:10:02.939953 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 10019
I20260501 02:10:02.945847 18819 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskQ5nb4m/build/debug/bin/kudu with pid 10096
2026-05-01T02:10:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3965 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7944 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (466777 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
1 FAILED TEST